+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210909/B1b --output-run DEEP/20210909/B1b/science#step0/20240317T070413Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 8.5139 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.191 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.5122 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.4647 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/pipeline_orig.qgraph DEEP/20210909/B1b/science#step0/20240317T070413Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.8267 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 8.9042 seconds; current memory usage: 0.270 Gibyte, delta: 0.092 Gibyte, peak delta: 0.079 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0133 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210909_B1b_science#step0_20240317T070413Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.4851 seconds; current memory usage: 0.270 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.270 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 1.7042 seconds; current memory usage: 0.272 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 3330d6b1-876c-4633-9b06-0ac7aee4e2fa parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 0 executors parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22732096912592 parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-17 07:04:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-17 07:04:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:04:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 07:04:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:04:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-17 07:04:58 proc_lsst.multi:170 [INFO] [multi] job_id 53768 proc_lsst.multi INFO: [multi] job_id 53768 2024-03-17 07:04:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:04:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:04:58 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 53768 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 53768 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 53768 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 18ad7b8cc8fd9b5da16045197fc7ca4a parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/f96c952d-2fbe-4982-a02c-d392e26d03b0_isrForCrosstalkSources_1031740_11.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/f96c952d-2fbe-4982-a02c-d392e26d03b0_isrForCrosstalkSources_1031740_11.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash e5ad1073b186b75a59b152aaaa0a7643 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/217b75f3-2d65-4b2d-ab00-b9aed604d290_isrForCrosstalkSources_1031739_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/217b75f3-2d65-4b2d-ab00-b9aed604d290_isrForCrosstalkSources_1031739_43.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 9ea9840f2aacb4615e14009ca8cb56b6 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b29880f6-13ec-4ded-93e4-b12057247e6f_isrForCrosstalkSources_1031740_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b29880f6-13ec-4ded-93e4-b12057247e6f_isrForCrosstalkSources_1031740_46.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash f4513bc30037542b0fa8fc7e53c39ae3 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d50a7ee5-a4b9-42a9-9862-1abbc7a77a88_isrForCrosstalkSources_1031739_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d50a7ee5-a4b9-42a9-9862-1abbc7a77a88_isrForCrosstalkSources_1031739_21.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 55d1cac1cc2e1f8a1fc0c928c6f64161 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/5dac5f8b-248f-4da7-bd43-c44e3a7d1eba_isrForCrosstalkSources_1031740_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/5dac5f8b-248f-4da7-bd43-c44e3a7d1eba_isrForCrosstalkSources_1031740_51.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 740e674c28669db85d51e496737a4279 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5d2c3c76-1144-4600-97ed-6d8de7b781f3_isrForCrosstalkSources_1031741_22.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5d2c3c76-1144-4600-97ed-6d8de7b781f3_isrForCrosstalkSources_1031741_22.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 0a8f8350c909c65187a02291beb66c80 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/94469cc5-1b5d-46f1-8dfb-3aed8302d667_isrForCrosstalkSources_1031740_39.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/94469cc5-1b5d-46f1-8dfb-3aed8302d667_isrForCrosstalkSources_1031740_39.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 896942ef23f0983c0d6c8b8d589dd607 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/036638ea-b4a4-46c2-aa8d-e755e8202403_isrForCrosstalkSources_1031739_13.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/036638ea-b4a4-46c2-aa8d-e755e8202403_isrForCrosstalkSources_1031739_13.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 6b532012e79c4a3f1f444a1619bb9a59 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/03861dac-a142-488e-b098-316ac515e5a5_isrForCrosstalkSources_1031741_61.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/03861dac-a142-488e-b098-316ac515e5a5_isrForCrosstalkSources_1031741_61.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 5e642ec225bd63b0ea6592a04bd7f005 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/937b7311-3133-423f-98d2-45f875fcc9fa_isrForCrosstalkSources_1031740_22.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/937b7311-3133-423f-98d2-45f875fcc9fa_isrForCrosstalkSources_1031740_22.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 2d3ec124a62e8fb71a93338364c94b9d parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/622958c7-4e95-44e0-80ef-69495156ccff_isrForCrosstalkSources_1031739_9.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/622958c7-4e95-44e0-80ef-69495156ccff_isrForCrosstalkSources_1031739_9.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 0f303699bc0f52ea8262d3ddee2c81e1 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba48bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c9378bf6-740e-441c-bd39-93d957498306_isrForCrosstalkSources_1031739_19.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c9378bf6-740e-441c-bd39-93d957498306_isrForCrosstalkSources_1031739_19.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 3f7ebc5d858628f48f1d0e40ab6bd0ba parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a6fb2bb4-f79c-446e-8a38-4f530c7ce883_isrForCrosstalkSources_1031740_4.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a6fb2bb4-f79c-446e-8a38-4f530c7ce883_isrForCrosstalkSources_1031740_4.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash f6cf953e8dd7fb6370c2e50d71b203f7 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2b318e3e-2bc4-47dc-b579-e57fe63ce331_isrForCrosstalkSources_1031741_42.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2b318e3e-2bc4-47dc-b579-e57fe63ce331_isrForCrosstalkSources_1031741_42.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 96a93986c4a125ca33ca25c4fd5c49ba parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/25dc0e8c-85c0-4eea-a163-0e3a1759e8c5_isrForCrosstalkSources_1031739_35.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/25dc0e8c-85c0-4eea-a163-0e3a1759e8c5_isrForCrosstalkSources_1031739_35.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash f51ecc169b38b7f2375ab798ea62eb8a parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8d10fa7c-5069-45b4-8ce8-c38b680aaa5d_isrForCrosstalkSources_1031741_9.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8d10fa7c-5069-45b4-8ce8-c38b680aaa5d_isrForCrosstalkSources_1031741_9.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash caaf122ad91a1a5a2da2b271ae2bd8f4 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c4028811-ff51-425e-b4ff-b742e71dccff_isrForCrosstalkSources_1031739_10.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c4028811-ff51-425e-b4ff-b742e71dccff_isrForCrosstalkSources_1031739_10.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash e97a4be6e23fdd7ca9c77d1be00caf53 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5ac3363d-7877-46cd-a785-4d75338497a8_isrForCrosstalkSources_1031739_38.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5ac3363d-7877-46cd-a785-4d75338497a8_isrForCrosstalkSources_1031739_38.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 8007156d7af7b6e2336dfb8e052057bc parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6fd4ab88-add4-4024-b0b1-cd3d583dbeeb_isrForCrosstalkSources_1031740_57.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6fd4ab88-add4-4024-b0b1-cd3d583dbeeb_isrForCrosstalkSources_1031740_57.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 585691603216d551375d26cf0d918455 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ab74fd0-beee-4837-901f-aef21102964c_isrForCrosstalkSources_1031740_28.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ab74fd0-beee-4837-901f-aef21102964c_isrForCrosstalkSources_1031740_28.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 74b80979abe69c6843f9979402f9153d parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/72ec5f94-1884-4b9f-b8ca-12a78a67b914_isrForCrosstalkSources_1031741_41.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/72ec5f94-1884-4b9f-b8ca-12a78a67b914_isrForCrosstalkSources_1031741_41.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 1f4353547dec947a6d8bd81f407ae00d parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/51901269-0186-4a7b-a044-ea20b2ec4c3e_isrForCrosstalkSources_1031739_55.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/51901269-0186-4a7b-a044-ea20b2ec4c3e_isrForCrosstalkSources_1031739_55.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash c8b13c677d9e88a8f8f15ea644335390 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/eadcde68-0bd0-449e-95d0-41161e5e5b03_isrForCrosstalkSources_1031739_36.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/eadcde68-0bd0-449e-95d0-41161e5e5b03_isrForCrosstalkSources_1031739_36.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 5ea845c07378d163e6532f2e5a9108a1 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4e37c51d-89d9-40a7-a29d-66e3ad3388e7_isrForCrosstalkSources_1031739_16.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4e37c51d-89d9-40a7-a29d-66e3ad3388e7_isrForCrosstalkSources_1031739_16.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash b6741b8ad7c2ab0ad1cf8e847b3a754d parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/258351ca-e05c-4ddf-97e4-9e9139ee3532_isrForCrosstalkSources_1031741_16.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/258351ca-e05c-4ddf-97e4-9e9139ee3532_isrForCrosstalkSources_1031741_16.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 71f014734a7c1eca2b890b40cfede1dc parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/02ba45c9-b800-472f-ae51-5108c7f91cbf_isrForCrosstalkSources_1031741_51.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/02ba45c9-b800-472f-ae51-5108c7f91cbf_isrForCrosstalkSources_1031741_51.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash b9facb02487335dccf6a977bc1c2c480 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9f5b4348-3e32-4d2c-9305-88cf8c902eb2_isrForCrosstalkSources_1031739_1.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9f5b4348-3e32-4d2c-9305-88cf8c902eb2_isrForCrosstalkSources_1031739_1.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 4c27194eed3aafbc307440ad546adbbc parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/fd4cbd3e-92da-4383-a323-72f09407e5e2_isrForCrosstalkSources_1031741_59.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/fd4cbd3e-92da-4383-a323-72f09407e5e2_isrForCrosstalkSources_1031741_59.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 3cf086b6613c59f193a29dc13738b877 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9c6a0c67-842c-4719-a405-e85411f90d4b_isrForCrosstalkSources_1031739_11.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9c6a0c67-842c-4719-a405-e85411f90d4b_isrForCrosstalkSources_1031739_11.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 55153665323f11e7fe4c2022e5a6a260 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/fb879f7f-f7ff-4e4c-80fd-fd448719cb9d_isrForCrosstalkSources_1031740_30.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/fb879f7f-f7ff-4e4c-80fd-fd448719cb9d_isrForCrosstalkSources_1031740_30.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 30cfc949e741470e6dc264b0b0da3a23 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9d277ccf-28f6-43a5-9a8c-efa1e744d6ee_isrForCrosstalkSources_1031741_56.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9d277ccf-28f6-43a5-9a8c-efa1e744d6ee_isrForCrosstalkSources_1031741_56.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 3ea915367a1d746b4e321479fd3bb227 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7929e594-71d6-482f-9feb-9ac4a6f728a9_isrForCrosstalkSources_1031739_59.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7929e594-71d6-482f-9feb-9ac4a6f728a9_isrForCrosstalkSources_1031739_59.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 7230dbf8591de6cd649f32e813c666ac parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ea9d3407-e3ab-4a9a-9c6f-0b7f20e79335_isrForCrosstalkSources_1031741_43.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ea9d3407-e3ab-4a9a-9c6f-0b7f20e79335_isrForCrosstalkSources_1031741_43.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash af0bd62381ca6c0ac29b569f9de2b283 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a2d559c-b2eb-4800-b5c2-2f48af470c3e_isrForCrosstalkSources_1031740_1.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a2d559c-b2eb-4800-b5c2-2f48af470c3e_isrForCrosstalkSources_1031740_1.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 5981b650d772d32e5ffe4a0c123f4ee9 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/e5ca966e-58ea-45e2-9af8-3967813a44fa_isrForCrosstalkSources_1031740_24.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/e5ca966e-58ea-45e2-9af8-3967813a44fa_isrForCrosstalkSources_1031740_24.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash a2dfbea39d1e6dd4a0d56a2f422041e2 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/135317da-1b84-4592-8229-cf6b0ea95bb2_isrForCrosstalkSources_1031741_28.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/135317da-1b84-4592-8229-cf6b0ea95bb2_isrForCrosstalkSources_1031741_28.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash f6ab3a1c0661ab8015ffecc771eecec8 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/26e6e13a-c118-4dfc-81cb-b6c01383345d_isrForCrosstalkSources_1031740_9.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/26e6e13a-c118-4dfc-81cb-b6c01383345d_isrForCrosstalkSources_1031740_9.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 02c9fd3d747688e0ab5b21c01e6892c8 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c19beb9c-ab80-4fa2-9269-feb81139450b_isrForCrosstalkSources_1031740_7.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c19beb9c-ab80-4fa2-9269-feb81139450b_isrForCrosstalkSources_1031740_7.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 2a0502dd25e536d4a4fd7a5bb97d1a1f parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/075bd84a-0c7d-4647-adb4-81c181aff817_isrForCrosstalkSources_1031740_3.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/075bd84a-0c7d-4647-adb4-81c181aff817_isrForCrosstalkSources_1031740_3.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 2a9fdafd79411d27420d32270e846371 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b8686479-0f0c-46dc-ac36-47118eac1f49_isrForCrosstalkSources_1031740_14.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b8686479-0f0c-46dc-ac36-47118eac1f49_isrForCrosstalkSources_1031740_14.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 97df3ba2954c808c1252e01ccea21c2d parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/62cf6004-b36d-498d-bf93-c934a216b7dd_isrForCrosstalkSources_1031740_33.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/62cf6004-b36d-498d-bf93-c934a216b7dd_isrForCrosstalkSources_1031740_33.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash c83d467f11dcd37eb154a228cb691100 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3a0af342-d87b-4333-ada5-d505a8d9a8ba_isrForCrosstalkSources_1031740_20.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3a0af342-d87b-4333-ada5-d505a8d9a8ba_isrForCrosstalkSources_1031740_20.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash a090f730c2751535ef2aeee55db5439c parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c5781d02-525d-42f4-a4ae-e0557643fc8f_isrForCrosstalkSources_1031740_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c5781d02-525d-42f4-a4ae-e0557643fc8f_isrForCrosstalkSources_1031740_31.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 8fcf72244c91fd40307977f6a387e904 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9788e4b3-5753-4d34-a730-d09e1d94789d_isrForCrosstalkSources_1031740_32.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9788e4b3-5753-4d34-a730-d09e1d94789d_isrForCrosstalkSources_1031740_32.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 68d78e9bd2dfc1e64c2e0b0c33fdb91f parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5be33edd-4c6e-4be7-b616-177d1adf5917_isrForCrosstalkSources_1031741_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5be33edd-4c6e-4be7-b616-177d1adf5917_isrForCrosstalkSources_1031741_48.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 96b70ed274f1f4feaa22c7e1c1796cd2 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/4c4c1ab0-c5b7-4ee9-a4e9-7d49a1214f10_isrForCrosstalkSources_1031740_27.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/4c4c1ab0-c5b7-4ee9-a4e9-7d49a1214f10_isrForCrosstalkSources_1031740_27.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 5348137061af36b7c1b3b67e27a4bfc9 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/148a9178-43b7-48e9-9510-a4adb3d6871f_isrForCrosstalkSources_1031740_18.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/148a9178-43b7-48e9-9510-a4adb3d6871f_isrForCrosstalkSources_1031740_18.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 59740bafb3592b89b26fa8ba5b0a27e9 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/12a85342-67eb-40f0-b239-e204f6c840ea_isrForCrosstalkSources_1031740_13.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/12a85342-67eb-40f0-b239-e204f6c840ea_isrForCrosstalkSources_1031740_13.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash f9aed4779628db62bc7eeeeb6f3acee0 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b0ef8c95-b6bb-48cb-9856-9f154b5babf2_isrForCrosstalkSources_1031741_10.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b0ef8c95-b6bb-48cb-9856-9f154b5babf2_isrForCrosstalkSources_1031741_10.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash c6ac56157d76066e3642480469a3574b parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/65c6493c-ca14-4875-be55-64d87d200cb5_isrForCrosstalkSources_1031740_19.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/65c6493c-ca14-4875-be55-64d87d200cb5_isrForCrosstalkSources_1031740_19.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash d9ad55af0530abed3184f63e7f6e0b20 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ae539533-71c4-4ec8-91e1-f1135d321e11_isrForCrosstalkSources_1031741_1.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ae539533-71c4-4ec8-91e1-f1135d321e11_isrForCrosstalkSources_1031741_1.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash d501444f063b425d9cc1c074bd6af745 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/34baec3d-1c34-4234-b7c5-55b176644ed2_isrForCrosstalkSources_1031740_55.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/34baec3d-1c34-4234-b7c5-55b176644ed2_isrForCrosstalkSources_1031740_55.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 6aa8c6dac326b90f348dc14dc6d65f8d parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d4f458b6-43c2-4b7f-85ea-a6e69887a257_isrForCrosstalkSources_1031741_53.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d4f458b6-43c2-4b7f-85ea-a6e69887a257_isrForCrosstalkSources_1031741_53.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash c9d5d38d45e9f10fb49346fde7fcfd35 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4eb39acb-2d63-499c-8795-6d1e8f58bf92_isrForCrosstalkSources_1031741_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4eb39acb-2d63-499c-8795-6d1e8f58bf92_isrForCrosstalkSources_1031741_2.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash c1bf1d6986d70369513ff17ee0b3623b parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/95a9b941-0516-441b-9362-f0f1cc2a2789_isrForCrosstalkSources_1031740_37.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/95a9b941-0516-441b-9362-f0f1cc2a2789_isrForCrosstalkSources_1031740_37.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 0f042b25e0e4e0b649e394f4fdb23a37 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a0c600f-8705-4664-ac2f-3a9def889542_isrForCrosstalkSources_1031739_8.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a0c600f-8705-4664-ac2f-3a9def889542_isrForCrosstalkSources_1031739_8.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 7d16bae70551c784507b5803857e5fdf parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/58cf7d69-31cb-4b3f-808d-cca6a29e967d_isrForCrosstalkSources_1031740_43.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/58cf7d69-31cb-4b3f-808d-cca6a29e967d_isrForCrosstalkSources_1031740_43.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 948547ea7f01b159965250b270543f58 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d8592fee-dba6-4af3-8f03-712a414178cc_isrForCrosstalkSources_1031741_29.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d8592fee-dba6-4af3-8f03-712a414178cc_isrForCrosstalkSources_1031741_29.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 7d7cdeeccb11f30a9d77284de84574d6 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ad8bb238-8558-4c07-a6d6-433aec486c36_isrForCrosstalkSources_1031739_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ad8bb238-8558-4c07-a6d6-433aec486c36_isrForCrosstalkSources_1031739_18.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash f62706ff1b0016a09acd1f7f0781adaf parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/35296e50-2b02-4832-aeb5-dcd9d4916905_isrForCrosstalkSources_1031739_56.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/35296e50-2b02-4832-aeb5-dcd9d4916905_isrForCrosstalkSources_1031739_56.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 35170fd6f56cf39e77f8419d438f5e42 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6a44be86-2bec-4ef9-98f6-6fed45754cc2_isrForCrosstalkSources_1031740_49.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6a44be86-2bec-4ef9-98f6-6fed45754cc2_isrForCrosstalkSources_1031740_49.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 9f10a4df54b5829c20b37c446b606f47 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7c09cc37-8b09-4fc5-8f10-eb9310ac4061_isrForCrosstalkSources_1031740_40.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7c09cc37-8b09-4fc5-8f10-eb9310ac4061_isrForCrosstalkSources_1031740_40.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 2ba6c7125e0eb97fadc09e36f6a7c087 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/28619702-0f8c-43f9-93b7-2490345d14d8_isrForCrosstalkSources_1031740_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/28619702-0f8c-43f9-93b7-2490345d14d8_isrForCrosstalkSources_1031740_60.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash ea6c74100fb8048ede6d10da82e15d99 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/db8e9bc2-1ccb-436e-9346-1ffd8140d904_isrForCrosstalkSources_1031740_47.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/db8e9bc2-1ccb-436e-9346-1ffd8140d904_isrForCrosstalkSources_1031740_47.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 000da5d445a851661624f3620e8503cb parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3363a383-4456-44a0-940f-97ffa601cda1_isrForCrosstalkSources_1031740_8.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3363a383-4456-44a0-940f-97ffa601cda1_isrForCrosstalkSources_1031740_8.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash bcdd1d1b90ededf202c102fc754b6622 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/717e4d3a-b62b-407c-8a23-b75eee05b6b2_isrForCrosstalkSources_1031739_40.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/717e4d3a-b62b-407c-8a23-b75eee05b6b2_isrForCrosstalkSources_1031739_40.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash a178af1cafc364d403638f8062e67710 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5f5bed90-6557-44fe-b4d2-fda6c8774eac_isrForCrosstalkSources_1031739_53.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5f5bed90-6557-44fe-b4d2-fda6c8774eac_isrForCrosstalkSources_1031739_53.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 406aff61d17607fa24ab50089f234579 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/b83caf70-dc04-4021-9389-c59ba7056981_isrForCrosstalkSources_1031739_14.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/b83caf70-dc04-4021-9389-c59ba7056981_isrForCrosstalkSources_1031739_14.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 3974d58ec1dd5f183bdf76d733eb2604 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ddfa89ae-d3d9-4753-ac97-50a04915c5ec_isrForCrosstalkSources_1031739_41.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ddfa89ae-d3d9-4753-ac97-50a04915c5ec_isrForCrosstalkSources_1031739_41.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 2de268bee9e82da27459243db9271696 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/777fa7e8-ff55-42d7-b2ec-2e72c6ab8bec_isrForCrosstalkSources_1031741_5.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/777fa7e8-ff55-42d7-b2ec-2e72c6ab8bec_isrForCrosstalkSources_1031741_5.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 17fd28dacd59e597646368df8bd4affa parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/524a2ac7-0a4f-41c2-a8be-e9d92db6ac9f_isrForCrosstalkSources_1031741_11.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/524a2ac7-0a4f-41c2-a8be-e9d92db6ac9f_isrForCrosstalkSources_1031741_11.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 83d6f49fbacb399022654dad9df7176f parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d3816ee3-d071-4c15-b074-70b5680b93e8_isrForCrosstalkSources_1031741_12.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d3816ee3-d071-4c15-b074-70b5680b93e8_isrForCrosstalkSources_1031741_12.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash c7aa1540c99964c1f5d2b9e74d70fb7d parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4c06aa77-d4dc-4fb3-94f2-d0c34da09b1f_isrForCrosstalkSources_1031739_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4c06aa77-d4dc-4fb3-94f2-d0c34da09b1f_isrForCrosstalkSources_1031739_2.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 18107e2a69d503a840abc9cfc5bb1029 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/49d83469-84d0-4e2f-8b40-ba76ca5b6741_isrForCrosstalkSources_1031740_15.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/49d83469-84d0-4e2f-8b40-ba76ca5b6741_isrForCrosstalkSources_1031740_15.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash c7144173c0b3a4ad554cd569a8f8da94 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ceac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d97f9e95-0a62-42bd-b0af-b8bc01a792b0_isrForCrosstalkSources_1031739_49.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d97f9e95-0a62-42bd-b0af-b8bc01a792b0_isrForCrosstalkSources_1031739_49.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 4e605b3896da5d5df54b3e4bc40400af parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ceb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d4c9155-ac5e-4906-a936-618cef6cfa98_isrForCrosstalkSources_1031739_12.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d4c9155-ac5e-4906-a936-618cef6cfa98_isrForCrosstalkSources_1031739_12.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash cf2ee3bb51939396c6ecd6de8fd25e77 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ceca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/96fabf67-7c02-4f7c-b40b-8ebf6dbbbecd_isrForCrosstalkSources_1031741_25.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/96fabf67-7c02-4f7c-b40b-8ebf6dbbbecd_isrForCrosstalkSources_1031741_25.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash f3c5b8671818f4a81246b22a287edb53 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bdaa7435-cb11-4355-946b-e2518fb377ff_isrForCrosstalkSources_1031740_36.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bdaa7435-cb11-4355-946b-e2518fb377ff_isrForCrosstalkSources_1031740_36.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash a118dceebaa08d3915e75571cfe7afe7 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4ced40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/67e9741e-44ce-45b3-a956-25141b58a725_isrForCrosstalkSources_1031739_22.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/67e9741e-44ce-45b3-a956-25141b58a725_isrForCrosstalkSources_1031739_22.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 500b55c6a522fee860bd105b5ef0aa0d parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0db44751-b139-4f71-b560-7e523ebd374f_isrForCrosstalkSources_1031741_35.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0db44751-b139-4f71-b560-7e523ebd374f_isrForCrosstalkSources_1031741_35.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash bc935e0e632f43841e8726fc445bbdd6 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/71f9e55b-92b6-4372-bcec-f4b35ffea530_isrForCrosstalkSources_1031739_4.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/71f9e55b-92b6-4372-bcec-f4b35ffea530_isrForCrosstalkSources_1031739_4.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash be8cb4a63c877bed059bf011aacd242d parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/da41a460-29ee-4702-81f2-0cdcd4db3298_isrForCrosstalkSources_1031741_37.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/da41a460-29ee-4702-81f2-0cdcd4db3298_isrForCrosstalkSources_1031741_37.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 4290e25727421889ba50c1416aab256d parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/523fd647-1618-48a9-a554-0be344b2878a_isrForCrosstalkSources_1031739_54.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/523fd647-1618-48a9-a554-0be344b2878a_isrForCrosstalkSources_1031739_54.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash e07f3a46030245bea6ff08a05f9fb82f parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3c57a682-51a4-476a-a917-f8b5bc280389_isrForCrosstalkSources_1031740_38.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3c57a682-51a4-476a-a917-f8b5bc280389_isrForCrosstalkSources_1031740_38.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 351692820d1caf3458c3c7d0ef1d7b42 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/44ce442e-f95f-43a5-8d17-16c3396a2bb1_isrForCrosstalkSources_1031739_62.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/44ce442e-f95f-43a5-8d17-16c3396a2bb1_isrForCrosstalkSources_1031739_62.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 5d79543c7f3800a2b1fb7c1fc63c68af parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8e7be0da-3408-4bfc-bc0a-c94578075ce6_isrForCrosstalkSources_1031741_45.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8e7be0da-3408-4bfc-bc0a-c94578075ce6_isrForCrosstalkSources_1031741_45.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash f4ea0c389bda48b3bfb0dca73ca3ac72 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/feec20c6-8052-4576-bbf2-426bccb7aef8_isrForCrosstalkSources_1031739_15.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/feec20c6-8052-4576-bbf2-426bccb7aef8_isrForCrosstalkSources_1031739_15.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash e0ea660bd77d705938718a8a3f50d4d2 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/85b9ea55-c7ff-443c-bcb6-858c32e76eec_isrForCrosstalkSources_1031741_62.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/85b9ea55-c7ff-443c-bcb6-858c32e76eec_isrForCrosstalkSources_1031741_62.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 7140adb7f499b9e60763dd0564b96f1d parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7fbcac33-8831-495a-8367-791ccfbe85ff_isrForCrosstalkSources_1031740_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7fbcac33-8831-495a-8367-791ccfbe85ff_isrForCrosstalkSources_1031740_2.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 6c456121e0db42abfa323e0609673692 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/93a7d841-910e-4ba0-a2ef-0eeb4c3645c9_isrForCrosstalkSources_1031739_17.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/93a7d841-910e-4ba0-a2ef-0eeb4c3645c9_isrForCrosstalkSources_1031739_17.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 32e1bcea6383d262c69a2288742583bf parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ec4838eb-bf55-41b6-8496-97c7cd8b090b_isrForCrosstalkSources_1031741_32.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ec4838eb-bf55-41b6-8496-97c7cd8b090b_isrForCrosstalkSources_1031741_32.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 0657319cb16b698e427056fb721d9316 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/3c079371-b0a5-4588-aab5-31aa6bedbcaf_isrForCrosstalkSources_1031741_46.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/3c079371-b0a5-4588-aab5-31aa6bedbcaf_isrForCrosstalkSources_1031741_46.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 5cb97a6568bd96cd0ab535044fdb3dcc parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/f7f1e9b8-68fd-4c0a-9af9-e3406eaeb473_isrForCrosstalkSources_1031739_23.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/f7f1e9b8-68fd-4c0a-9af9-e3406eaeb473_isrForCrosstalkSources_1031739_23.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash d85fb0e33864e126a70bc54130536b3b parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6ec77f19-86ae-4ec4-8830-f4ff3c05aac7_isrForCrosstalkSources_1031740_25.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6ec77f19-86ae-4ec4-8830-f4ff3c05aac7_isrForCrosstalkSources_1031740_25.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash afbfb363de48a1b1379d99463f39ec6a parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/13155e70-3a83-42fd-b8ba-085b12acff16_isrForCrosstalkSources_1031741_31.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/13155e70-3a83-42fd-b8ba-085b12acff16_isrForCrosstalkSources_1031741_31.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 274e088b2c73f723cd47c77b12107bab parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5c75147e-578f-454c-ab4b-73669d5d128c_isrForCrosstalkSources_1031741_33.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5c75147e-578f-454c-ab4b-73669d5d128c_isrForCrosstalkSources_1031741_33.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 84edfe84771477f678ad9de7d6b61797 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/922a1aa9-f54f-49e9-ac8b-a3b80d7d0798_isrForCrosstalkSources_1031740_58.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/922a1aa9-f54f-49e9-ac8b-a3b80d7d0798_isrForCrosstalkSources_1031740_58.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 1c9bd6d0fb0c629dd5c21e882486d14e parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b9152656-45b8-4c15-8058-53c672315833_isrForCrosstalkSources_1031741_40.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b9152656-45b8-4c15-8058-53c672315833_isrForCrosstalkSources_1031741_40.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 7da5a6ea010d98a30809244138fa246e parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ed59d68c-d792-41da-abb7-ea32cf91dffe_isrForCrosstalkSources_1031739_44.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ed59d68c-d792-41da-abb7-ea32cf91dffe_isrForCrosstalkSources_1031739_44.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 8d6e9f3ce36c71400bf1d14a76541ffb parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/373b8166-cc75-4d35-b538-6fe8e4715d52_isrForCrosstalkSources_1031740_54.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/373b8166-cc75-4d35-b538-6fe8e4715d52_isrForCrosstalkSources_1031740_54.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 46a6317276cb4673b3c363daaf68916a parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c8a54e8f-c3bd-4007-a9d9-c695bf255aa3_isrForCrosstalkSources_1031739_45.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c8a54e8f-c3bd-4007-a9d9-c695bf255aa3_isrForCrosstalkSources_1031739_45.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash dab1031e58651cd5dfbbeb4e0131c0d8 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ceee8d91-7811-4f7c-9f99-5431ac83b0fc_isrForCrosstalkSources_1031741_52.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ceee8d91-7811-4f7c-9f99-5431ac83b0fc_isrForCrosstalkSources_1031741_52.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 13d38618716f948dc4a9559132553e65 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3ae490c0-a171-4d43-9689-cc6076a6ceeb_isrForCrosstalkSources_1031739_5.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3ae490c0-a171-4d43-9689-cc6076a6ceeb_isrForCrosstalkSources_1031739_5.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash b595c22df5fbbe8b349dd35db673a844 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcd5bc4a-4fa9-41f6-855a-cd72d9982533_isrForCrosstalkSources_1031741_49.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcd5bc4a-4fa9-41f6-855a-cd72d9982533_isrForCrosstalkSources_1031741_49.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash c110c106866feb5bf9c3ccb8d56a0545 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0c582cc6-f930-4778-8a92-edc012015558_isrForCrosstalkSources_1031741_27.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0c582cc6-f930-4778-8a92-edc012015558_isrForCrosstalkSources_1031741_27.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 1763ce61e7196451f13d56ce4b629a47 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/212cc9e2-c1c2-4318-9234-460446c55a54_isrForCrosstalkSources_1031739_3.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/212cc9e2-c1c2-4318-9234-460446c55a54_isrForCrosstalkSources_1031739_3.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 62d3dc94f39a741dae94b55fdbc266e3 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/88200989-e95a-469b-ba8f-370777ddf620_isrForCrosstalkSources_1031739_42.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/88200989-e95a-469b-ba8f-370777ddf620_isrForCrosstalkSources_1031739_42.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash da396158757e4fb8ec62fd16384cc97c parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acba4cc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/97021850-6416-41fe-842c-7f3c1481290c_isrForCrosstalkSources_1031741_47.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/97021850-6416-41fe-842c-7f3c1481290c_isrForCrosstalkSources_1031741_47.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash dd763568fe041badf7f7d4f0fea04e1b parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f200e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7d71c92b-2fc6-4676-9e5d-b0a9ca076152_isrForCrosstalkSources_1031739_51.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7d71c92b-2fc6-4676-9e5d-b0a9ca076152_isrForCrosstalkSources_1031739_51.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 82d1505fa9edb2d83c8532ba63852d22 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/13dac28a-c16d-418a-bc4d-2b1814ceeecc_isrForCrosstalkSources_1031740_50.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/13dac28a-c16d-418a-bc4d-2b1814ceeecc_isrForCrosstalkSources_1031740_50.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash fda67ef2a43aa5f41fcc546e4372e1f1 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f202c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bab6ca0c-c448-4121-b11c-03afe26285f6_isrForCrosstalkSources_1031741_26.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bab6ca0c-c448-4121-b11c-03afe26285f6_isrForCrosstalkSources_1031741_26.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 2d2ed4817291bdf18fe5dc395053ebf3 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/42e590f9-b9be-4305-9443-fc30071040f6_isrForCrosstalkSources_1031739_30.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/42e590f9-b9be-4305-9443-fc30071040f6_isrForCrosstalkSources_1031739_30.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash c1dce9ab43f63dfd2f7b8276650e4182 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/04983e43-887b-407c-a5d3-d977d7c0854f_isrForCrosstalkSources_1031740_10.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/04983e43-887b-407c-a5d3-d977d7c0854f_isrForCrosstalkSources_1031740_10.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 3c16208407d11e8d920b3866cd07c92f parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/baf1b6b3-524d-4bc7-ae87-4db7ccc740cd_isrForCrosstalkSources_1031741_60.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/baf1b6b3-524d-4bc7-ae87-4db7ccc740cd_isrForCrosstalkSources_1031741_60.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 38334b4841e6669bbddcf44d6b30245b parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f204a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/2162ef67-5e9c-4f8b-bc28-41fc42de66d9_isrForCrosstalkSources_1031740_21.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/2162ef67-5e9c-4f8b-bc28-41fc42de66d9_isrForCrosstalkSources_1031740_21.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash b2408a13faa91162dc7b7b202ae6012b parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f205e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e61e2d9f-f4e7-4516-b2ed-8f2a10963e52_isrForCrosstalkSources_1031739_61.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e61e2d9f-f4e7-4516-b2ed-8f2a10963e52_isrForCrosstalkSources_1031739_61.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 7556028dae25ccddc5356022abb5ee3b parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/275c803d-6cbb-4aee-9245-5344c4b64dc8_isrForCrosstalkSources_1031740_62.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/275c803d-6cbb-4aee-9245-5344c4b64dc8_isrForCrosstalkSources_1031740_62.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 5919811922432f138295063317bc1a36 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a5dc967-da2e-40b8-9d37-5cf6f06fed83_isrForCrosstalkSources_1031740_17.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a5dc967-da2e-40b8-9d37-5cf6f06fed83_isrForCrosstalkSources_1031740_17.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 02a6458209c0dd65a9e15707f5a450b4 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9efedbe4-9367-4404-97bd-46ff65d6a302_isrForCrosstalkSources_1031739_57.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9efedbe4-9367-4404-97bd-46ff65d6a302_isrForCrosstalkSources_1031739_57.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 295c080be7d61444d22cb6dec1b734d1 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f207c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/033c75a1-ff76-4983-9a36-154f13eacb32_isrForCrosstalkSources_1031741_54.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/033c75a1-ff76-4983-9a36-154f13eacb32_isrForCrosstalkSources_1031741_54.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash e2d9d91c9ecb3f06be29be301d660b9a parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d2f4dae0-76f1-4e7a-90a7-0109a1a73747_isrForCrosstalkSources_1031739_29.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d2f4dae0-76f1-4e7a-90a7-0109a1a73747_isrForCrosstalkSources_1031739_29.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash c6e4d1fce03a6ad47aae7f6450690d56 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0b8355ee-39db-4bcc-b982-5bba89b536e9_isrForCrosstalkSources_1031739_37.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0b8355ee-39db-4bcc-b982-5bba89b536e9_isrForCrosstalkSources_1031739_37.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 3c9ada7b822297cba5bf60c5a5b69d69 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f209a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/927f969d-dd65-4d2b-ab79-224d3404d9d5_isrForCrosstalkSources_1031740_23.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/927f969d-dd65-4d2b-ab79-224d3404d9d5_isrForCrosstalkSources_1031740_23.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash a321fa4043fe99978542baeb15defb94 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/246df2ce-466f-4a7a-a0ce-7fca17a7937b_isrForCrosstalkSources_1031739_60.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/246df2ce-466f-4a7a-a0ce-7fca17a7937b_isrForCrosstalkSources_1031739_60.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash a5ea9c44548dd7ea3712f84fa702840f parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/22a7e1cc-ea41-4bae-8651-820f3ae1ebde_isrForCrosstalkSources_1031739_47.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/22a7e1cc-ea41-4bae-8651-820f3ae1ebde_isrForCrosstalkSources_1031739_47.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash c224f370c3cadb710d46504ca013a59c parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/98a9a5ec-5e81-43b9-b012-0993b786eb67_isrForCrosstalkSources_1031741_57.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/98a9a5ec-5e81-43b9-b012-0993b786eb67_isrForCrosstalkSources_1031741_57.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 355f284b46e4c5b846c4f9c259af2276 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3c214be0-c7de-4d64-9b5e-58cc58855427_isrForCrosstalkSources_1031739_58.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3c214be0-c7de-4d64-9b5e-58cc58855427_isrForCrosstalkSources_1031739_58.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 3d95ee633689aed6e3acbdcd2718750b parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/887b308b-d200-4ff9-8328-9eb3a95dd56c_isrForCrosstalkSources_1031740_26.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/887b308b-d200-4ff9-8328-9eb3a95dd56c_isrForCrosstalkSources_1031740_26.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 16038ff4d49957605a08510660db97d6 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/6e32e895-c634-49e5-83ad-7f0a67178c17_isrForCrosstalkSources_1031741_8.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/6e32e895-c634-49e5-83ad-7f0a67178c17_isrForCrosstalkSources_1031741_8.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 60ebf9c74bfc507ab1d1dc167b714814 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/70e79934-0aba-43bf-80c0-7317e264b548_isrForCrosstalkSources_1031739_26.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/70e79934-0aba-43bf-80c0-7317e264b548_isrForCrosstalkSources_1031739_26.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 1882980b1b1edf47e126e3bc412e79be parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bfcc0c8e-7d16-431d-ad2d-e72bdee8f248_isrForCrosstalkSources_1031740_45.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bfcc0c8e-7d16-431d-ad2d-e72bdee8f248_isrForCrosstalkSources_1031740_45.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 883e41641094f39b656080deb6a2702f parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/58bea049-75c7-49c7-ba75-ebd7f450e113_isrForCrosstalkSources_1031739_52.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/58bea049-75c7-49c7-ba75-ebd7f450e113_isrForCrosstalkSources_1031739_52.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash f710c1b0f0cc8037b8a7671aebae6391 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f20fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9e87fb25-9678-41eb-9e84-eba2004385e6_isrForCrosstalkSources_1031740_48.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9e87fb25-9678-41eb-9e84-eba2004385e6_isrForCrosstalkSources_1031740_48.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 51c5761db455aee99756f5cf2df05fd5 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a51ce11a-b645-4d86-af84-183534281512_isrForCrosstalkSources_1031740_56.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a51ce11a-b645-4d86-af84-183534281512_isrForCrosstalkSources_1031740_56.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash c577fbfee86641b4b3fd83e291b79503 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b6f3cca8-1a71-42d9-b322-344a31fd0078_isrForCrosstalkSources_1031741_21.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b6f3cca8-1a71-42d9-b322-344a31fd0078_isrForCrosstalkSources_1031741_21.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 55659f55cb0031e1c3b8b6b452fed010 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/cee77076-7350-420c-ba5a-03640c79fcda_isrForCrosstalkSources_1031740_44.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/cee77076-7350-420c-ba5a-03640c79fcda_isrForCrosstalkSources_1031740_44.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 87a599071d15a3b5aee33c89498ad7c9 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f211c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/26415ca8-b126-4dba-9c60-adbe9933619a_isrForCrosstalkSources_1031739_6.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/26415ca8-b126-4dba-9c60-adbe9933619a_isrForCrosstalkSources_1031739_6.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 36e4293f685552e3bb36c12a8b242761 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4ad3deac-751f-4837-8093-b15fdfa857ab_isrForCrosstalkSources_1031741_6.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4ad3deac-751f-4837-8093-b15fdfa857ab_isrForCrosstalkSources_1031741_6.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 2727561c4be1cf066dce1118b51a0ba2 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f213a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ecf40eec-8669-48c4-a129-ace4dc33e91c_isrForCrosstalkSources_1031741_36.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ecf40eec-8669-48c4-a129-ace4dc33e91c_isrForCrosstalkSources_1031741_36.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 1c85995e5567d46468184e37563caf27 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a987bc78-a60c-4ae9-9508-f2f4f7034409_isrForCrosstalkSources_1031741_39.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a987bc78-a60c-4ae9-9508-f2f4f7034409_isrForCrosstalkSources_1031741_39.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 744220a376ee8e305983eae876488d7c parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/a3d3136c-4886-4e3c-b36a-0e58f9f9c572_isrForCrosstalkSources_1031739_33.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/a3d3136c-4886-4e3c-b36a-0e58f9f9c572_isrForCrosstalkSources_1031739_33.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash f3a48999cec90ad47b292bf15cc60d3c parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f214e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a8da8dff-c3c5-4a87-83ab-ec5ea23e591f_isrForCrosstalkSources_1031741_15.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a8da8dff-c3c5-4a87-83ab-ec5ea23e591f_isrForCrosstalkSources_1031741_15.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 7008824c6b8dce5d47a6b9c5a70af84b parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0485b3bc-9d74-4e31-bd0d-a6c5fbad03f9_isrForCrosstalkSources_1031741_14.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0485b3bc-9d74-4e31-bd0d-a6c5fbad03f9_isrForCrosstalkSources_1031741_14.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 249b8acf0e2809b579ba0fcf58d5e51a parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f216c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/600198e2-33e3-4363-88de-cb291d13b25a_isrForCrosstalkSources_1031740_34.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/600198e2-33e3-4363-88de-cb291d13b25a_isrForCrosstalkSources_1031740_34.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash c66379569d20e67643fb498a60584268 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/231e7c16-9a6a-49ad-9592-df5362379c1b_isrForCrosstalkSources_1031741_3.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/231e7c16-9a6a-49ad-9592-df5362379c1b_isrForCrosstalkSources_1031741_3.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 2d70733011c0285c742bca386cc7f851 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f218a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aa64a890-ec0e-4552-91c2-a14c9db406d2_isrForCrosstalkSources_1031739_28.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aa64a890-ec0e-4552-91c2-a14c9db406d2_isrForCrosstalkSources_1031739_28.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash de0d397d7494b9db33395c5d194f44df parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5ca6369d-6cae-4f3e-9374-165ad857b986_isrForCrosstalkSources_1031741_34.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5ca6369d-6cae-4f3e-9374-165ad857b986_isrForCrosstalkSources_1031741_34.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 2956e592eeba46f2def1edbf9bd18aff parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0cfbf8a6-15c5-465a-8341-f96cef7dd4bb_isrForCrosstalkSources_1031739_31.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0cfbf8a6-15c5-465a-8341-f96cef7dd4bb_isrForCrosstalkSources_1031739_31.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash dd795448dc45f467d7696fc48c9318cd parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f219e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bf6a07a1-1364-42c2-b871-60095813774e_isrForCrosstalkSources_1031741_24.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bf6a07a1-1364-42c2-b871-60095813774e_isrForCrosstalkSources_1031741_24.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 09830bc5ebdd0c6de82a4430d9b8691c parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7577a54c-25b0-432d-a4c0-8de016e5be09_isrForCrosstalkSources_1031740_59.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7577a54c-25b0-432d-a4c0-8de016e5be09_isrForCrosstalkSources_1031740_59.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 52718b7cf4f3d290ef5c482ed5635cd4 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ac78428-4e49-40ae-817b-d0d8aca51601_isrForCrosstalkSources_1031740_29.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ac78428-4e49-40ae-817b-d0d8aca51601_isrForCrosstalkSources_1031740_29.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash d28685d92f2f1bb25a10706784e0c469 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aebe1a92-c13a-48e8-a0b4-c271ffedb42b_isrForCrosstalkSources_1031739_34.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aebe1a92-c13a-48e8-a0b4-c271ffedb42b_isrForCrosstalkSources_1031739_34.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 43665893e8f50e1264ce1fcdddbba0c8 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/de53b6f1-e05b-4196-916c-ec4575991ab2_isrForCrosstalkSources_1031741_20.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/de53b6f1-e05b-4196-916c-ec4575991ab2_isrForCrosstalkSources_1031741_20.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 9d4174f74f1ce58a4da51ceda4f8aa15 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/625561d9-199c-483c-a508-21b9b79775a7_isrForCrosstalkSources_1031740_6.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/625561d9-199c-483c-a508-21b9b79775a7_isrForCrosstalkSources_1031740_6.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 3d59a7a571e4f97fd7eac34dc7f148dc parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cdd9fb65-640f-4675-bcd9-c6c729b801c8_isrForCrosstalkSources_1031741_18.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cdd9fb65-640f-4675-bcd9-c6c729b801c8_isrForCrosstalkSources_1031741_18.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash c02c147d47c6fc84e03176c4a0f41501 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b64c3e17-3759-4fe9-8041-9bd23c08d83a_isrForCrosstalkSources_1031741_58.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b64c3e17-3759-4fe9-8041-9bd23c08d83a_isrForCrosstalkSources_1031741_58.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 255eb383ced93abe1a349a6f34e471fd parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/acfbb67c-9dce-4c8b-8cef-b52fc3cc3d2c_isrForCrosstalkSources_1031741_38.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/acfbb67c-9dce-4c8b-8cef-b52fc3cc3d2c_isrForCrosstalkSources_1031741_38.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash bedb776a6035a5f5e40fd941cf26ac02 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f21f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/057526d1-9792-4f59-84ad-1276b0b73ae9_isrForCrosstalkSources_1031741_17.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/057526d1-9792-4f59-84ad-1276b0b73ae9_isrForCrosstalkSources_1031741_17.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash b7d2f0fdacc845e8a8eae6fa84c06946 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/6e0eea99-eb7b-4d11-82b5-c1c8c8764511_isrForCrosstalkSources_1031739_27.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/6e0eea99-eb7b-4d11-82b5-c1c8c8764511_isrForCrosstalkSources_1031739_27.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash c0bbdae80e62d8847c9db4a0d1f6ee3c parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f220c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/f6eefef9-cf01-433e-8609-ed0c0c644788_isrForCrosstalkSources_1031741_23.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/f6eefef9-cf01-433e-8609-ed0c0c644788_isrForCrosstalkSources_1031741_23.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash ef689de3f9197f9831b487808f47ffd5 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/24b4118b-539c-45b9-bf68-741dcf0697e2_isrForCrosstalkSources_1031740_5.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/24b4118b-539c-45b9-bf68-741dcf0697e2_isrForCrosstalkSources_1031740_5.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 6fd54ebcd4658e34b9cb9f072e6426b3 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/95d7a0c3-3d3a-4c4d-980c-6ebaeb101f27_isrForCrosstalkSources_1031739_48.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/95d7a0c3-3d3a-4c4d-980c-6ebaeb101f27_isrForCrosstalkSources_1031739_48.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash b5b8a458bea0965e7b848e82b05bbf3a parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f222a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cbb811e4-be87-4073-bbcb-a613325593ec_isrForCrosstalkSources_1031741_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cbb811e4-be87-4073-bbcb-a613325593ec_isrForCrosstalkSources_1031741_55.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 5e14d12ae54b46481e2872ebb1dcb1c8 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4bc2bebf-1823-4431-9266-c83692fd4e24_isrForCrosstalkSources_1031739_46.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4bc2bebf-1823-4431-9266-c83692fd4e24_isrForCrosstalkSources_1031739_46.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 53bef407340a7ae7d0b4c8a87fe797ab parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f223e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9bab137d-50cc-4028-b5f0-ea6bf40db059_isrForCrosstalkSources_1031741_30.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9bab137d-50cc-4028-b5f0-ea6bf40db059_isrForCrosstalkSources_1031741_30.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 320773265bdbcfd2c28ae3b34fbc3d5b parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/1bacc129-4bb0-4c11-b52d-09d1f3b03b31_isrForCrosstalkSources_1031740_61.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/1bacc129-4bb0-4c11-b52d-09d1f3b03b31_isrForCrosstalkSources_1031740_61.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash c72d2582d39333f38157066c28d4fbc9 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a2636c8d-2797-4d13-a63b-c841f9567100_isrForCrosstalkSources_1031740_16.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a2636c8d-2797-4d13-a63b-c841f9567100_isrForCrosstalkSources_1031740_16.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 3add3798ebd76020d1102f45640062b8 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f225c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a3cd226-1d91-41e6-b08e-1c78f0a0286c_isrForCrosstalkSources_1031739_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a3cd226-1d91-41e6-b08e-1c78f0a0286c_isrForCrosstalkSources_1031739_25.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 03927ac4183536c93e25d2350b3516bc parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/76263766-8721-4510-ae31-8828fb5f3723_isrForCrosstalkSources_1031740_41.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/76263766-8721-4510-ae31-8828fb5f3723_isrForCrosstalkSources_1031740_41.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 96ded85bcb67685c4e7d1fd4b9bb6e0b parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e40b2ad9-01ee-4289-94f4-38bc18011548_isrForCrosstalkSources_1031739_24.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e40b2ad9-01ee-4289-94f4-38bc18011548_isrForCrosstalkSources_1031739_24.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 30ac1c6ed3bfa6ed11ab3b2a09fd5af2 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a181f316-17bc-4a00-8e33-c1b4d6e05ba6_isrForCrosstalkSources_1031740_52.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a181f316-17bc-4a00-8e33-c1b4d6e05ba6_isrForCrosstalkSources_1031740_52.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash b7e07062ecb7f5db9d5190169a097ca8 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f227a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2a4202aa-77a9-47fd-a166-7a18fecc5de7_isrForCrosstalkSources_1031741_4.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2a4202aa-77a9-47fd-a166-7a18fecc5de7_isrForCrosstalkSources_1031741_4.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash b3498fdcd6c81885b4d1a4ff93440efa parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f228e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a39552b6-53d3-41c8-bfa7-f13de5b27572_isrForCrosstalkSources_1031741_50.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a39552b6-53d3-41c8-bfa7-f13de5b27572_isrForCrosstalkSources_1031741_50.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 4c7f39f5d73d7d7734d7c47c1e9a6565 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcaf6bb1-c3b8-40e4-84ad-557e9a094286_isrForCrosstalkSources_1031741_13.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcaf6bb1-c3b8-40e4-84ad-557e9a094286_isrForCrosstalkSources_1031741_13.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 7cc8b2bf4f04a5dbc55c4c86db65916b parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/cc8d7399-8f84-4b90-81b4-5d109e890b2b_isrForCrosstalkSources_1031739_50.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/cc8d7399-8f84-4b90-81b4-5d109e890b2b_isrForCrosstalkSources_1031739_50.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 9b38aaeea92efe67e69c5dd127250c30 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c478d5b2-e757-4d99-b571-efab00dbdce7_isrForCrosstalkSources_1031740_12.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c478d5b2-e757-4d99-b571-efab00dbdce7_isrForCrosstalkSources_1031740_12.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 3bccfc2bfadb0021d291e8562bc29933 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/99378acb-95e5-4fd8-b5bb-9b555dcaecfb_isrForCrosstalkSources_1031740_53.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/99378acb-95e5-4fd8-b5bb-9b555dcaecfb_isrForCrosstalkSources_1031740_53.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash c44b3b17700688a821de9efaaaaf82e7 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b7113960-cb4f-4b57-a64b-601623ca22fd_isrForCrosstalkSources_1031740_35.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b7113960-cb4f-4b57-a64b-601623ca22fd_isrForCrosstalkSources_1031740_35.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash c3ae27b61a09975fa0fc422e76190101 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d1bd05e8-7df8-40c0-85c7-b5d0c0d617b3_isrForCrosstalkSources_1031741_7.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d1bd05e8-7df8-40c0-85c7-b5d0c0d617b3_isrForCrosstalkSources_1031741_7.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash ebc3ea3eaf9c4346bd61de0ea929a0be parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/35e2256c-22d2-4a01-89f8-f5bcaa4875d8_isrForCrosstalkSources_1031741_44.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/35e2256c-22d2-4a01-89f8-f5bcaa4875d8_isrForCrosstalkSources_1031741_44.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash defe9f117c2cafd0168b5ed655b5e439 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d9f3371-b706-4bb3-a28b-1739445a084e_isrForCrosstalkSources_1031739_39.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d9f3371-b706-4bb3-a28b-1739445a084e_isrForCrosstalkSources_1031739_39.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 70316cb6f90f91df5522549dfdbc3849 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/09bc09b1-8f5c-46bf-aeb2-6dd6de3bf1f4_isrForCrosstalkSources_1031739_32.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/09bc09b1-8f5c-46bf-aeb2-6dd6de3bf1f4_isrForCrosstalkSources_1031739_32.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 5dc25155371fff8384abdd4c4de6f39d parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/97630da0-37c1-4041-ada7-9984360fe5c2_isrForCrosstalkSources_1031739_7.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/97630da0-37c1-4041-ada7-9984360fe5c2_isrForCrosstalkSources_1031739_7.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 6dcaf7fb1791e18c85a7c5735dc3ecf9 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f22fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/32d97d2a-71c5-4d3b-9304-a41d922e3c0d_isrForCrosstalkSources_1031741_19.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/32d97d2a-71c5-4d3b-9304-a41d922e3c0d_isrForCrosstalkSources_1031741_19.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash f217298404c3e850c6337ba5a4eac539 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f23060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/99fbf786-5be6-4f19-95fa-5e0385bd9306_isrForCrosstalkSources_1031739_20.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/99fbf786-5be6-4f19-95fa-5e0385bd9306_isrForCrosstalkSources_1031739_20.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 9c64337766f127a353b3616b3fd7f981 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14acb8f231a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/69a2c0be-235d-49d9-8a18-dd977e83cd17_isrForCrosstalkSources_1031740_42.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/69a2c0be-235d-49d9-8a18-dd977e83cd17_isrForCrosstalkSources_1031740_42.stderr 2024-03-17 07:05:02 proc_lsst.multi:146 [INFO] found job 53768 in provider local proc_lsst.multi INFO: found job 53768 in provider local parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 186 parsl.dataflow.strategy DEBUG: Executor multi has 186 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 46 more blocks parsl.executors.status_handling INFO: Scaling out by 46 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-17 07:05:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-17 07:05:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 07:05:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:04 proc_lsst.multi:170 [INFO] [multi] job_id 17153088 proc_lsst.multi INFO: [multi] job_id 17153088 2024-03-17 07:05:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153088 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153088 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17153088 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-17 07:05:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-17 07:05:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:04 proc_lsst.multi:170 [INFO] [multi] job_id 17153089 proc_lsst.multi INFO: [multi] job_id 17153089 2024-03-17 07:05:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 07:05:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153089 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153089 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17153089 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-17 07:05:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-17 07:05:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 07:05:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:05 proc_lsst.multi:170 [INFO] [multi] job_id 17153090 proc_lsst.multi INFO: [multi] job_id 17153090 2024-03-17 07:05:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 07:05:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153090 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153090 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17153090 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-17 07:05:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-17 07:05:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 07:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:05 proc_lsst.multi:170 [INFO] [multi] job_id 17153091 proc_lsst.multi INFO: [multi] job_id 17153091 2024-03-17 07:05:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 07:05:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153091 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153091 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17153091 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-17 07:05:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-17 07:05:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 07:05:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:06 proc_lsst.multi:170 [INFO] [multi] job_id 17153093 proc_lsst.multi INFO: [multi] job_id 17153093 2024-03-17 07:05:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153093 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153093 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17153093 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-17 07:05:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-17 07:05:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:06 proc_lsst.multi:170 [INFO] [multi] job_id 17153094 proc_lsst.multi INFO: [multi] job_id 17153094 2024-03-17 07:05:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 07:05:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153094 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153094 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17153094 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-17 07:05:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-17 07:05:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 07:05:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:07 proc_lsst.multi:170 [INFO] [multi] job_id 17153096 proc_lsst.multi INFO: [multi] job_id 17153096 2024-03-17 07:05:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 07:05:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153096 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153096 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17153096 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-17 07:05:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-17 07:05:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 07:05:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:08 proc_lsst.multi:170 [INFO] [multi] job_id 17153097 proc_lsst.multi INFO: [multi] job_id 17153097 2024-03-17 07:05:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 07:05:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153097 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153097 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17153097 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-17 07:05:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-17 07:05:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 07:05:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:10 proc_lsst.multi:170 [INFO] [multi] job_id 17153098 proc_lsst.multi INFO: [multi] job_id 17153098 2024-03-17 07:05:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 07:05:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153098 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153098 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17153098 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-17 07:05:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-17 07:05:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 07:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:10 proc_lsst.multi:170 [INFO] [multi] job_id 17153099 proc_lsst.multi INFO: [multi] job_id 17153099 2024-03-17 07:05:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 07:05:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153099 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153099 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17153099 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-17 07:05:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-17 07:05:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 07:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:10 proc_lsst.multi:170 [INFO] [multi] job_id 17153100 proc_lsst.multi INFO: [multi] job_id 17153100 2024-03-17 07:05:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 07:05:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153100 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153100 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17153100 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-17 07:05:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-17 07:05:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 07:05:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:11 proc_lsst.multi:170 [INFO] [multi] job_id 17153101 proc_lsst.multi INFO: [multi] job_id 17153101 2024-03-17 07:05:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 07:05:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153101 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153101 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17153101 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-17 07:05:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-17 07:05:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 07:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:11 proc_lsst.multi:170 [INFO] [multi] job_id 17153103 proc_lsst.multi INFO: [multi] job_id 17153103 2024-03-17 07:05:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 07:05:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153103 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153103 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17153103 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-17 07:05:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-17 07:05:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 07:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:11 proc_lsst.multi:170 [INFO] [multi] job_id 17153104 proc_lsst.multi INFO: [multi] job_id 17153104 2024-03-17 07:05:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 07:05:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153104 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153104 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17153104 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-17 07:05:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-17 07:05:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 07:05:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:12 proc_lsst.multi:170 [INFO] [multi] job_id 17153106 proc_lsst.multi INFO: [multi] job_id 17153106 2024-03-17 07:05:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 07:05:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153106 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153106 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17153106 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-17 07:05:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-17 07:05:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 07:05:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:13 proc_lsst.multi:170 [INFO] [multi] job_id 17153108 proc_lsst.multi INFO: [multi] job_id 17153108 2024-03-17 07:05:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 07:05:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153108 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153108 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17153108 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-17 07:05:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-17 07:05:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 07:05:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:14 proc_lsst.multi:170 [INFO] [multi] job_id 17153110 proc_lsst.multi INFO: [multi] job_id 17153110 2024-03-17 07:05:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 07:05:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153110 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153110 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17153110 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-17 07:05:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-17 07:05:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 07:05:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:15 proc_lsst.multi:170 [INFO] [multi] job_id 17153115 proc_lsst.multi INFO: [multi] job_id 17153115 2024-03-17 07:05:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 07:05:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153115 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153115 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17153115 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-17 07:05:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-17 07:05:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 07:05:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:16 proc_lsst.multi:170 [INFO] [multi] job_id 17153117 proc_lsst.multi INFO: [multi] job_id 17153117 2024-03-17 07:05:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 07:05:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153117 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153117 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17153117 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-17 07:05:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-17 07:05:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 07:05:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:17 proc_lsst.multi:170 [INFO] [multi] job_id 17153118 proc_lsst.multi INFO: [multi] job_id 17153118 2024-03-17 07:05:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 07:05:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153118 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153118 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17153118 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-17 07:05:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-17 07:05:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 07:05:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18ad7b8cc8fd9b5da16045197fc7ca4a with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/f96c952d-2fbe-4982-a02c-d392e26d03b0_isrForCrosstalkSources_1031740_11.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/f96c952d-2fbe-4982-a02c-d392e26d03b0_isrForCrosstalkSources_1031740_11.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5ad1073b186b75a59b152aaaa0a7643 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/217b75f3-2d65-4b2d-ab00-b9aed604d290_isrForCrosstalkSources_1031739_43.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/217b75f3-2d65-4b2d-ab00-b9aed604d290_isrForCrosstalkSources_1031739_43.stderr 2024-03-17 07:05:18 proc_lsst.multi:170 [INFO] [multi] job_id 17153119 proc_lsst.multi INFO: [multi] job_id 17153119 2024-03-17 07:05:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 07:05:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153119 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153119 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17153119 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-17 07:05:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-17 07:05:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 07:05:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:19 proc_lsst.multi:170 [INFO] [multi] job_id 17153120 proc_lsst.multi INFO: [multi] job_id 17153120 2024-03-17 07:05:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 07:05:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153120 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153120 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17153120 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-17 07:05:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-17 07:05:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 07:05:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:20 proc_lsst.multi:170 [INFO] [multi] job_id 17153121 proc_lsst.multi INFO: [multi] job_id 17153121 2024-03-17 07:05:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 07:05:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153121 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153121 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17153121 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-17 07:05:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-17 07:05:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 07:05:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:21 proc_lsst.multi:170 [INFO] [multi] job_id 17153123 proc_lsst.multi INFO: [multi] job_id 17153123 2024-03-17 07:05:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 07:05:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153123 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153123 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17153123 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-17 07:05:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-17 07:05:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 07:05:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:22 proc_lsst.multi:170 [INFO] [multi] job_id 17153125 proc_lsst.multi INFO: [multi] job_id 17153125 2024-03-17 07:05:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 07:05:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153125 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153125 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17153125 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-17 07:05:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-17 07:05:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 07:05:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:23 proc_lsst.multi:170 [INFO] [multi] job_id 17153127 proc_lsst.multi INFO: [multi] job_id 17153127 2024-03-17 07:05:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 07:05:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153127 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153127 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17153127 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-17 07:05:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-17 07:05:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 07:05:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ea9840f2aacb4615e14009ca8cb56b6 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b29880f6-13ec-4ded-93e4-b12057247e6f_isrForCrosstalkSources_1031740_46.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b29880f6-13ec-4ded-93e4-b12057247e6f_isrForCrosstalkSources_1031740_46.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a8f8350c909c65187a02291beb66c80 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/94469cc5-1b5d-46f1-8dfb-3aed8302d667_isrForCrosstalkSources_1031740_39.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/94469cc5-1b5d-46f1-8dfb-3aed8302d667_isrForCrosstalkSources_1031740_39.stderr 2024-03-17 07:05:24 proc_lsst.multi:170 [INFO] [multi] job_id 17153129 proc_lsst.multi INFO: [multi] job_id 17153129 2024-03-17 07:05:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 07:05:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153129 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153129 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17153129 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-17 07:05:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-17 07:05:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 07:05:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4513bc30037542b0fa8fc7e53c39ae3 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d50a7ee5-a4b9-42a9-9862-1abbc7a77a88_isrForCrosstalkSources_1031739_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d50a7ee5-a4b9-42a9-9862-1abbc7a77a88_isrForCrosstalkSources_1031739_21.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55d1cac1cc2e1f8a1fc0c928c6f64161 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/5dac5f8b-248f-4da7-bd43-c44e3a7d1eba_isrForCrosstalkSources_1031740_51.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/5dac5f8b-248f-4da7-bd43-c44e3a7d1eba_isrForCrosstalkSources_1031740_51.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 896942ef23f0983c0d6c8b8d589dd607 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/036638ea-b4a4-46c2-aa8d-e755e8202403_isrForCrosstalkSources_1031739_13.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/036638ea-b4a4-46c2-aa8d-e755e8202403_isrForCrosstalkSources_1031739_13.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 740e674c28669db85d51e496737a4279 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5d2c3c76-1144-4600-97ed-6d8de7b781f3_isrForCrosstalkSources_1031741_22.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5d2c3c76-1144-4600-97ed-6d8de7b781f3_isrForCrosstalkSources_1031741_22.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b532012e79c4a3f1f444a1619bb9a59 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/03861dac-a142-488e-b098-316ac515e5a5_isrForCrosstalkSources_1031741_61.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/03861dac-a142-488e-b098-316ac515e5a5_isrForCrosstalkSources_1031741_61.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e642ec225bd63b0ea6592a04bd7f005 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/937b7311-3133-423f-98d2-45f875fcc9fa_isrForCrosstalkSources_1031740_22.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/937b7311-3133-423f-98d2-45f875fcc9fa_isrForCrosstalkSources_1031740_22.stderr 2024-03-17 07:05:27 proc_lsst.multi:170 [INFO] [multi] job_id 17153135 proc_lsst.multi INFO: [multi] job_id 17153135 2024-03-17 07:05:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 07:05:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153135 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153135 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17153135 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-17 07:05:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-17 07:05:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 07:05:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f303699bc0f52ea8262d3ddee2c81e1 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c9378bf6-740e-441c-bd39-93d957498306_isrForCrosstalkSources_1031739_19.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c9378bf6-740e-441c-bd39-93d957498306_isrForCrosstalkSources_1031739_19.stderr 2024-03-17 07:05:28 proc_lsst.multi:170 [INFO] [multi] job_id 17153137 proc_lsst.multi INFO: [multi] job_id 17153137 2024-03-17 07:05:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 07:05:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153137 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153137 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17153137 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-17 07:05:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-17 07:05:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 07:05:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d3ec124a62e8fb71a93338364c94b9d with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/622958c7-4e95-44e0-80ef-69495156ccff_isrForCrosstalkSources_1031739_9.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/622958c7-4e95-44e0-80ef-69495156ccff_isrForCrosstalkSources_1031739_9.stderr 2024-03-17 07:05:30 proc_lsst.multi:170 [INFO] [multi] job_id 17153140 proc_lsst.multi INFO: [multi] job_id 17153140 2024-03-17 07:05:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:30 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17153140 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17153140 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17153140 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-17 07:05:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-17 07:05:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 07:05:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f7ebc5d858628f48f1d0e40ab6bd0ba with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a6fb2bb4-f79c-446e-8a38-4f530c7ce883_isrForCrosstalkSources_1031740_4.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a6fb2bb4-f79c-446e-8a38-4f530c7ce883_isrForCrosstalkSources_1031740_4.stderr 2024-03-17 07:05:31 proc_lsst.multi:170 [INFO] [multi] job_id 17153142 proc_lsst.multi INFO: [multi] job_id 17153142 2024-03-17 07:05:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17153142 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17153142 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17153142 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-17 07:05:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-17 07:05:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:33 proc_lsst.multi:170 [INFO] [multi] job_id 17153145 proc_lsst.multi INFO: [multi] job_id 17153145 2024-03-17 07:05:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 07:05:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:33 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17153145 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17153145 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17153145 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-17 07:05:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-17 07:05:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 07:05:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cf953e8dd7fb6370c2e50d71b203f7 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2b318e3e-2bc4-47dc-b579-e57fe63ce331_isrForCrosstalkSources_1031741_42.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2b318e3e-2bc4-47dc-b579-e57fe63ce331_isrForCrosstalkSources_1031741_42.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f51ecc169b38b7f2375ab798ea62eb8a with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8d10fa7c-5069-45b4-8ce8-c38b680aaa5d_isrForCrosstalkSources_1031741_9.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8d10fa7c-5069-45b4-8ce8-c38b680aaa5d_isrForCrosstalkSources_1031741_9.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a93986c4a125ca33ca25c4fd5c49ba with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/25dc0e8c-85c0-4eea-a163-0e3a1759e8c5_isrForCrosstalkSources_1031739_35.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/25dc0e8c-85c0-4eea-a163-0e3a1759e8c5_isrForCrosstalkSources_1031739_35.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caaf122ad91a1a5a2da2b271ae2bd8f4 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c4028811-ff51-425e-b4ff-b742e71dccff_isrForCrosstalkSources_1031739_10.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c4028811-ff51-425e-b4ff-b742e71dccff_isrForCrosstalkSources_1031739_10.stderr 2024-03-17 07:05:35 proc_lsst.multi:170 [INFO] [multi] job_id 17153146 proc_lsst.multi INFO: [multi] job_id 17153146 2024-03-17 07:05:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 07:05:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:35 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17153146 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17153146 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17153146 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-17 07:05:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-17 07:05:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 07:05:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8007156d7af7b6e2336dfb8e052057bc with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6fd4ab88-add4-4024-b0b1-cd3d583dbeeb_isrForCrosstalkSources_1031740_57.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6fd4ab88-add4-4024-b0b1-cd3d583dbeeb_isrForCrosstalkSources_1031740_57.stderr 2024-03-17 07:05:36 proc_lsst.multi:170 [INFO] [multi] job_id 17153149 proc_lsst.multi INFO: [multi] job_id 17153149 2024-03-17 07:05:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 07:05:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:36 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17153149 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17153149 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17153149 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-17 07:05:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-17 07:05:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 07:05:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e97a4be6e23fdd7ca9c77d1be00caf53 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5ac3363d-7877-46cd-a785-4d75338497a8_isrForCrosstalkSources_1031739_38.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5ac3363d-7877-46cd-a785-4d75338497a8_isrForCrosstalkSources_1031739_38.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cf086b6613c59f193a29dc13738b877 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9c6a0c67-842c-4719-a405-e85411f90d4b_isrForCrosstalkSources_1031739_11.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9c6a0c67-842c-4719-a405-e85411f90d4b_isrForCrosstalkSources_1031739_11.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c27194eed3aafbc307440ad546adbbc with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/fd4cbd3e-92da-4383-a323-72f09407e5e2_isrForCrosstalkSources_1031741_59.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/fd4cbd3e-92da-4383-a323-72f09407e5e2_isrForCrosstalkSources_1031741_59.stderr 2024-03-17 07:05:37 proc_lsst.multi:170 [INFO] [multi] job_id 17153151 proc_lsst.multi INFO: [multi] job_id 17153151 2024-03-17 07:05:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:37 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17153151 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17153151 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17153151 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-17 07:05:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-17 07:05:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 07:05:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:38 proc_lsst.multi:170 [INFO] [multi] job_id 17153153 proc_lsst.multi INFO: [multi] job_id 17153153 2024-03-17 07:05:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:38 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153153 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153153 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17153153 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-17 07:05:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-17 07:05:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:39 proc_lsst.multi:170 [INFO] [multi] job_id 17153155 proc_lsst.multi INFO: [multi] job_id 17153155 2024-03-17 07:05:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 07:05:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:39 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153155 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153155 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17153155 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-17 07:05:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-17 07:05:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 07:05:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:40 proc_lsst.multi:170 [INFO] [multi] job_id 17153157 proc_lsst.multi INFO: [multi] job_id 17153157 2024-03-17 07:05:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 07:05:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:40 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153157 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153157 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17153157 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-17 07:05:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-17 07:05:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 07:05:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a9fdafd79411d27420d32270e846371 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b8686479-0f0c-46dc-ac36-47118eac1f49_isrForCrosstalkSources_1031740_14.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b8686479-0f0c-46dc-ac36-47118eac1f49_isrForCrosstalkSources_1031740_14.stderr 2024-03-17 07:05:41 proc_lsst.multi:170 [INFO] [multi] job_id 17153159 proc_lsst.multi INFO: [multi] job_id 17153159 2024-03-17 07:05:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 07:05:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:41 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153159 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153159 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17153159 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-17 07:05:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-17 07:05:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 07:05:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55153665323f11e7fe4c2022e5a6a260 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/fb879f7f-f7ff-4e4c-80fd-fd448719cb9d_isrForCrosstalkSources_1031740_30.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/fb879f7f-f7ff-4e4c-80fd-fd448719cb9d_isrForCrosstalkSources_1031740_30.stderr 2024-03-17 07:05:42 proc_lsst.multi:170 [INFO] [multi] job_id 17153161 proc_lsst.multi INFO: [multi] job_id 17153161 2024-03-17 07:05:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:42 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153161 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153161 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17153161 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-17 07:05:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-17 07:05:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30cfc949e741470e6dc264b0b0da3a23 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9d277ccf-28f6-43a5-9a8c-efa1e744d6ee_isrForCrosstalkSources_1031741_56.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9d277ccf-28f6-43a5-9a8c-efa1e744d6ee_isrForCrosstalkSources_1031741_56.stderr 2024-03-17 07:05:43 proc_lsst.multi:170 [INFO] [multi] job_id 17153163 proc_lsst.multi INFO: [multi] job_id 17153163 2024-03-17 07:05:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 07:05:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:43 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153163 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153163 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17153163 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-17 07:05:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-17 07:05:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 07:05:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:44 proc_lsst.multi:170 [INFO] [multi] job_id 17153165 proc_lsst.multi INFO: [multi] job_id 17153165 2024-03-17 07:05:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 07:05:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:44 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153165 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153165 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17153165 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-17 07:05:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-17 07:05:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 07:05:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97df3ba2954c808c1252e01ccea21c2d with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/62cf6004-b36d-498d-bf93-c934a216b7dd_isrForCrosstalkSources_1031740_33.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/62cf6004-b36d-498d-bf93-c934a216b7dd_isrForCrosstalkSources_1031740_33.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c83d467f11dcd37eb154a228cb691100 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3a0af342-d87b-4333-ada5-d505a8d9a8ba_isrForCrosstalkSources_1031740_20.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3a0af342-d87b-4333-ada5-d505a8d9a8ba_isrForCrosstalkSources_1031740_20.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a090f730c2751535ef2aeee55db5439c with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c5781d02-525d-42f4-a4ae-e0557643fc8f_isrForCrosstalkSources_1031740_31.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c5781d02-525d-42f4-a4ae-e0557643fc8f_isrForCrosstalkSources_1031740_31.stderr 2024-03-17 07:05:45 proc_lsst.multi:170 [INFO] [multi] job_id 17153167 proc_lsst.multi INFO: [multi] job_id 17153167 2024-03-17 07:05:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 07:05:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:45 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153167 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153167 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17153167 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-17 07:05:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-17 07:05:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 07:05:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fcf72244c91fd40307977f6a387e904 with result from task 43 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9788e4b3-5753-4d34-a730-d09e1d94789d_isrForCrosstalkSources_1031740_32.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9788e4b3-5753-4d34-a730-d09e1d94789d_isrForCrosstalkSources_1031740_32.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68d78e9bd2dfc1e64c2e0b0c33fdb91f with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5be33edd-4c6e-4be7-b616-177d1adf5917_isrForCrosstalkSources_1031741_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5be33edd-4c6e-4be7-b616-177d1adf5917_isrForCrosstalkSources_1031741_48.stderr 2024-03-17 07:05:46 proc_lsst.multi:170 [INFO] [multi] job_id 17153169 proc_lsst.multi INFO: [multi] job_id 17153169 2024-03-17 07:05:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 07:05:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:46 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153169 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153169 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17153169 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-17 07:05:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-17 07:05:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 07:05:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96b70ed274f1f4feaa22c7e1c1796cd2 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/4c4c1ab0-c5b7-4ee9-a4e9-7d49a1214f10_isrForCrosstalkSources_1031740_27.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/4c4c1ab0-c5b7-4ee9-a4e9-7d49a1214f10_isrForCrosstalkSources_1031740_27.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59740bafb3592b89b26fa8ba5b0a27e9 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/12a85342-67eb-40f0-b239-e204f6c840ea_isrForCrosstalkSources_1031740_13.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/12a85342-67eb-40f0-b239-e204f6c840ea_isrForCrosstalkSources_1031740_13.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5348137061af36b7c1b3b67e27a4bfc9 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/148a9178-43b7-48e9-9510-a4adb3d6871f_isrForCrosstalkSources_1031740_18.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/148a9178-43b7-48e9-9510-a4adb3d6871f_isrForCrosstalkSources_1031740_18.stderr 2024-03-17 07:05:47 proc_lsst.multi:170 [INFO] [multi] job_id 17153171 proc_lsst.multi INFO: [multi] job_id 17153171 2024-03-17 07:05:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 07:05:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:47 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153171 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153171 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17153171 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-17 07:05:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-17 07:05:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 07:05:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 07:05:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 07:05:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 07:05:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 07:05:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 07:05:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 07:05:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 07:05:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 07:05:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,10.64.65.9,198.48.92.26,127.0.0.1,n3009,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54706 --result_port=54757 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 07:05:48 proc_lsst.multi:170 [INFO] [multi] job_id 17153173 proc_lsst.multi INFO: [multi] job_id 17153173 2024-03-17 07:05:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 07:05:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 07:05:48 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17153173 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17153173 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17153173 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22732096912592 2024-03-17 07:05:48 proc_lsst.multi:146 [INFO] found job 53768 in provider local proc_lsst.multi INFO: found job 53768 in provider local 2024-03-17 07:05:48 proc_lsst.multi:146 [INFO] found job 17153088 in provider astro proc_lsst.multi INFO: found job 17153088 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:50 proc_lsst.multi:146 [INFO] found job 17153089 in provider astro proc_lsst.multi INFO: found job 17153089 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9aed4779628db62bc7eeeeb6f3acee0 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b0ef8c95-b6bb-48cb-9856-9f154b5babf2_isrForCrosstalkSources_1031741_10.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b0ef8c95-b6bb-48cb-9856-9f154b5babf2_isrForCrosstalkSources_1031741_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:51 proc_lsst.multi:146 [INFO] found job 17153090 in provider astro proc_lsst.multi INFO: found job 17153090 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f4353547dec947a6d8bd81f407ae00d with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/51901269-0186-4a7b-a044-ea20b2ec4c3e_isrForCrosstalkSources_1031739_55.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/51901269-0186-4a7b-a044-ea20b2ec4c3e_isrForCrosstalkSources_1031739_55.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea845c07378d163e6532f2e5a9108a1 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4e37c51d-89d9-40a7-a29d-66e3ad3388e7_isrForCrosstalkSources_1031739_16.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4e37c51d-89d9-40a7-a29d-66e3ad3388e7_isrForCrosstalkSources_1031739_16.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6741b8ad7c2ab0ad1cf8e847b3a754d with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/258351ca-e05c-4ddf-97e4-9e9139ee3532_isrForCrosstalkSources_1031741_16.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/258351ca-e05c-4ddf-97e4-9e9139ee3532_isrForCrosstalkSources_1031741_16.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9facb02487335dccf6a977bc1c2c480 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9f5b4348-3e32-4d2c-9305-88cf8c902eb2_isrForCrosstalkSources_1031739_1.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9f5b4348-3e32-4d2c-9305-88cf8c902eb2_isrForCrosstalkSources_1031739_1.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 585691603216d551375d26cf0d918455 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ab74fd0-beee-4837-901f-aef21102964c_isrForCrosstalkSources_1031740_28.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ab74fd0-beee-4837-901f-aef21102964c_isrForCrosstalkSources_1031740_28.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b13c677d9e88a8f8f15ea644335390 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/eadcde68-0bd0-449e-95d0-41161e5e5b03_isrForCrosstalkSources_1031739_36.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/eadcde68-0bd0-449e-95d0-41161e5e5b03_isrForCrosstalkSources_1031739_36.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71f014734a7c1eca2b890b40cfede1dc with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/02ba45c9-b800-472f-ae51-5108c7f91cbf_isrForCrosstalkSources_1031741_51.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/02ba45c9-b800-472f-ae51-5108c7f91cbf_isrForCrosstalkSources_1031741_51.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74b80979abe69c6843f9979402f9153d with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/72ec5f94-1884-4b9f-b8ca-12a78a67b914_isrForCrosstalkSources_1031741_41.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/72ec5f94-1884-4b9f-b8ca-12a78a67b914_isrForCrosstalkSources_1031741_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:52 proc_lsst.multi:146 [INFO] found job 17153091 in provider astro proc_lsst.multi INFO: found job 17153091 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6ac56157d76066e3642480469a3574b with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/65c6493c-ca14-4875-be55-64d87d200cb5_isrForCrosstalkSources_1031740_19.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/65c6493c-ca14-4875-be55-64d87d200cb5_isrForCrosstalkSources_1031740_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:53 proc_lsst.multi:146 [INFO] found job 17153093 in provider astro proc_lsst.multi INFO: found job 17153093 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9ad55af0530abed3184f63e7f6e0b20 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ae539533-71c4-4ec8-91e1-f1135d321e11_isrForCrosstalkSources_1031741_1.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ae539533-71c4-4ec8-91e1-f1135d321e11_isrForCrosstalkSources_1031741_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:54 proc_lsst.multi:146 [INFO] found job 17153094 in provider astro proc_lsst.multi INFO: found job 17153094 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d16bae70551c784507b5803857e5fdf with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/58cf7d69-31cb-4b3f-808d-cca6a29e967d_isrForCrosstalkSources_1031740_43.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/58cf7d69-31cb-4b3f-808d-cca6a29e967d_isrForCrosstalkSources_1031740_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:55 proc_lsst.multi:146 [INFO] found job 17153096 in provider astro proc_lsst.multi INFO: found job 17153096 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 948547ea7f01b159965250b270543f58 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d8592fee-dba6-4af3-8f03-712a414178cc_isrForCrosstalkSources_1031741_29.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d8592fee-dba6-4af3-8f03-712a414178cc_isrForCrosstalkSources_1031741_29.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d7cdeeccb11f30a9d77284de84574d6 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ad8bb238-8558-4c07-a6d6-433aec486c36_isrForCrosstalkSources_1031739_18.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ad8bb238-8558-4c07-a6d6-433aec486c36_isrForCrosstalkSources_1031739_18.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f62706ff1b0016a09acd1f7f0781adaf with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/35296e50-2b02-4832-aeb5-dcd9d4916905_isrForCrosstalkSources_1031739_56.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/35296e50-2b02-4832-aeb5-dcd9d4916905_isrForCrosstalkSources_1031739_56.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35170fd6f56cf39e77f8419d438f5e42 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6a44be86-2bec-4ef9-98f6-6fed45754cc2_isrForCrosstalkSources_1031740_49.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6a44be86-2bec-4ef9-98f6-6fed45754cc2_isrForCrosstalkSources_1031740_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:56 proc_lsst.multi:146 [INFO] found job 17153097 in provider astro proc_lsst.multi INFO: found job 17153097 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6ab3a1c0661ab8015ffecc771eecec8 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/26e6e13a-c118-4dfc-81cb-b6c01383345d_isrForCrosstalkSources_1031740_9.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/26e6e13a-c118-4dfc-81cb-b6c01383345d_isrForCrosstalkSources_1031740_9.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af0bd62381ca6c0ac29b569f9de2b283 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a2d559c-b2eb-4800-b5c2-2f48af470c3e_isrForCrosstalkSources_1031740_1.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a2d559c-b2eb-4800-b5c2-2f48af470c3e_isrForCrosstalkSources_1031740_1.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c9fd3d747688e0ab5b21c01e6892c8 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c19beb9c-ab80-4fa2-9269-feb81139450b_isrForCrosstalkSources_1031740_7.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c19beb9c-ab80-4fa2-9269-feb81139450b_isrForCrosstalkSources_1031740_7.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5981b650d772d32e5ffe4a0c123f4ee9 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/e5ca966e-58ea-45e2-9af8-3967813a44fa_isrForCrosstalkSources_1031740_24.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/e5ca966e-58ea-45e2-9af8-3967813a44fa_isrForCrosstalkSources_1031740_24.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7230dbf8591de6cd649f32e813c666ac with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ea9d3407-e3ab-4a9a-9c6f-0b7f20e79335_isrForCrosstalkSources_1031741_43.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ea9d3407-e3ab-4a9a-9c6f-0b7f20e79335_isrForCrosstalkSources_1031741_43.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a0502dd25e536d4a4fd7a5bb97d1a1f with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/075bd84a-0c7d-4647-adb4-81c181aff817_isrForCrosstalkSources_1031740_3.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/075bd84a-0c7d-4647-adb4-81c181aff817_isrForCrosstalkSources_1031740_3.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2dfbea39d1e6dd4a0d56a2f422041e2 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/135317da-1b84-4592-8229-cf6b0ea95bb2_isrForCrosstalkSources_1031741_28.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/135317da-1b84-4592-8229-cf6b0ea95bb2_isrForCrosstalkSources_1031741_28.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 406aff61d17607fa24ab50089f234579 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/b83caf70-dc04-4021-9389-c59ba7056981_isrForCrosstalkSources_1031739_14.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/b83caf70-dc04-4021-9389-c59ba7056981_isrForCrosstalkSources_1031739_14.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3974d58ec1dd5f183bdf76d733eb2604 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ddfa89ae-d3d9-4753-ac97-50a04915c5ec_isrForCrosstalkSources_1031739_41.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ddfa89ae-d3d9-4753-ac97-50a04915c5ec_isrForCrosstalkSources_1031739_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:57 proc_lsst.multi:146 [INFO] found job 17153098 in provider astro proc_lsst.multi INFO: found job 17153098 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a178af1cafc364d403638f8062e67710 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5f5bed90-6557-44fe-b4d2-fda6c8774eac_isrForCrosstalkSources_1031739_53.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/5f5bed90-6557-44fe-b4d2-fda6c8774eac_isrForCrosstalkSources_1031739_53.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea915367a1d746b4e321479fd3bb227 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7929e594-71d6-482f-9feb-9ac4a6f728a9_isrForCrosstalkSources_1031739_59.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7929e594-71d6-482f-9feb-9ac4a6f728a9_isrForCrosstalkSources_1031739_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:05:58 proc_lsst.multi:146 [INFO] found job 17153099 in provider astro proc_lsst.multi INFO: found job 17153099 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:00 proc_lsst.multi:146 [INFO] found job 17153100 in provider astro proc_lsst.multi INFO: found job 17153100 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17fd28dacd59e597646368df8bd4affa with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/524a2ac7-0a4f-41c2-a8be-e9d92db6ac9f_isrForCrosstalkSources_1031741_11.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/524a2ac7-0a4f-41c2-a8be-e9d92db6ac9f_isrForCrosstalkSources_1031741_11.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf2ee3bb51939396c6ecd6de8fd25e77 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/96fabf67-7c02-4f7c-b40b-8ebf6dbbbecd_isrForCrosstalkSources_1031741_25.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/96fabf67-7c02-4f7c-b40b-8ebf6dbbbecd_isrForCrosstalkSources_1031741_25.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2de268bee9e82da27459243db9271696 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/777fa7e8-ff55-42d7-b2ec-2e72c6ab8bec_isrForCrosstalkSources_1031741_5.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/777fa7e8-ff55-42d7-b2ec-2e72c6ab8bec_isrForCrosstalkSources_1031741_5.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d6f49fbacb399022654dad9df7176f with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d3816ee3-d071-4c15-b074-70b5680b93e8_isrForCrosstalkSources_1031741_12.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d3816ee3-d071-4c15-b074-70b5680b93e8_isrForCrosstalkSources_1031741_12.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e605b3896da5d5df54b3e4bc40400af with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d4c9155-ac5e-4906-a936-618cef6cfa98_isrForCrosstalkSources_1031739_12.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d4c9155-ac5e-4906-a936-618cef6cfa98_isrForCrosstalkSources_1031739_12.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18107e2a69d503a840abc9cfc5bb1029 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/49d83469-84d0-4e2f-8b40-ba76ca5b6741_isrForCrosstalkSources_1031740_15.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/49d83469-84d0-4e2f-8b40-ba76ca5b6741_isrForCrosstalkSources_1031740_15.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7aa1540c99964c1f5d2b9e74d70fb7d with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4c06aa77-d4dc-4fb3-94f2-d0c34da09b1f_isrForCrosstalkSources_1031739_2.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4c06aa77-d4dc-4fb3-94f2-d0c34da09b1f_isrForCrosstalkSources_1031739_2.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7144173c0b3a4ad554cd569a8f8da94 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d97f9e95-0a62-42bd-b0af-b8bc01a792b0_isrForCrosstalkSources_1031739_49.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d97f9e95-0a62-42bd-b0af-b8bc01a792b0_isrForCrosstalkSources_1031739_49.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3c5b8671818f4a81246b22a287edb53 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bdaa7435-cb11-4355-946b-e2518fb377ff_isrForCrosstalkSources_1031740_36.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bdaa7435-cb11-4355-946b-e2518fb377ff_isrForCrosstalkSources_1031740_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:02 proc_lsst.multi:146 [INFO] found job 17153101 in provider astro proc_lsst.multi INFO: found job 17153101 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:04 proc_lsst.multi:146 [INFO] found job 17153103 in provider astro proc_lsst.multi INFO: found job 17153103 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb97a6568bd96cd0ab535044fdb3dcc with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/f7f1e9b8-68fd-4c0a-9af9-e3406eaeb473_isrForCrosstalkSources_1031739_23.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/f7f1e9b8-68fd-4c0a-9af9-e3406eaeb473_isrForCrosstalkSources_1031739_23.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afbfb363de48a1b1379d99463f39ec6a with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/13155e70-3a83-42fd-b8ba-085b12acff16_isrForCrosstalkSources_1031741_31.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/13155e70-3a83-42fd-b8ba-085b12acff16_isrForCrosstalkSources_1031741_31.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e07f3a46030245bea6ff08a05f9fb82f with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3c57a682-51a4-476a-a917-f8b5bc280389_isrForCrosstalkSources_1031740_38.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3c57a682-51a4-476a-a917-f8b5bc280389_isrForCrosstalkSources_1031740_38.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d85fb0e33864e126a70bc54130536b3b with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6ec77f19-86ae-4ec4-8830-f4ff3c05aac7_isrForCrosstalkSources_1031740_25.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/6ec77f19-86ae-4ec4-8830-f4ff3c05aac7_isrForCrosstalkSources_1031740_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:05 proc_lsst.multi:146 [INFO] found job 17153104 in provider astro proc_lsst.multi INFO: found job 17153104 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 274e088b2c73f723cd47c77b12107bab with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5c75147e-578f-454c-ab4b-73669d5d128c_isrForCrosstalkSources_1031741_33.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5c75147e-578f-454c-ab4b-73669d5d128c_isrForCrosstalkSources_1031741_33.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46a6317276cb4673b3c363daaf68916a with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c8a54e8f-c3bd-4007-a9d9-c695bf255aa3_isrForCrosstalkSources_1031739_45.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/c8a54e8f-c3bd-4007-a9d9-c695bf255aa3_isrForCrosstalkSources_1031739_45.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7da5a6ea010d98a30809244138fa246e with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ed59d68c-d792-41da-abb7-ea32cf91dffe_isrForCrosstalkSources_1031739_44.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/ed59d68c-d792-41da-abb7-ea32cf91dffe_isrForCrosstalkSources_1031739_44.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c9bd6d0fb0c629dd5c21e882486d14e with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b9152656-45b8-4c15-8058-53c672315833_isrForCrosstalkSources_1031741_40.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b9152656-45b8-4c15-8058-53c672315833_isrForCrosstalkSources_1031741_40.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d6e9f3ce36c71400bf1d14a76541ffb with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/373b8166-cc75-4d35-b538-6fe8e4715d52_isrForCrosstalkSources_1031740_54.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/373b8166-cc75-4d35-b538-6fe8e4715d52_isrForCrosstalkSources_1031740_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:06 proc_lsst.multi:146 [INFO] found job 17153106 in provider astro proc_lsst.multi INFO: found job 17153106 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13d38618716f948dc4a9559132553e65 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3ae490c0-a171-4d43-9689-cc6076a6ceeb_isrForCrosstalkSources_1031739_5.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3ae490c0-a171-4d43-9689-cc6076a6ceeb_isrForCrosstalkSources_1031739_5.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84edfe84771477f678ad9de7d6b61797 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/922a1aa9-f54f-49e9-ac8b-a3b80d7d0798_isrForCrosstalkSources_1031740_58.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/922a1aa9-f54f-49e9-ac8b-a3b80d7d0798_isrForCrosstalkSources_1031740_58.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62d3dc94f39a741dae94b55fdbc266e3 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/88200989-e95a-469b-ba8f-370777ddf620_isrForCrosstalkSources_1031739_42.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/88200989-e95a-469b-ba8f-370777ddf620_isrForCrosstalkSources_1031739_42.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 500b55c6a522fee860bd105b5ef0aa0d with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0db44751-b139-4f71-b560-7e523ebd374f_isrForCrosstalkSources_1031741_35.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0db44751-b139-4f71-b560-7e523ebd374f_isrForCrosstalkSources_1031741_35.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a118dceebaa08d3915e75571cfe7afe7 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/67e9741e-44ce-45b3-a956-25141b58a725_isrForCrosstalkSources_1031739_22.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/67e9741e-44ce-45b3-a956-25141b58a725_isrForCrosstalkSources_1031739_22.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c110c106866feb5bf9c3ccb8d56a0545 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0c582cc6-f930-4778-8a92-edc012015558_isrForCrosstalkSources_1031741_27.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0c582cc6-f930-4778-8a92-edc012015558_isrForCrosstalkSources_1031741_27.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dab1031e58651cd5dfbbeb4e0131c0d8 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ceee8d91-7811-4f7c-9f99-5431ac83b0fc_isrForCrosstalkSources_1031741_52.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ceee8d91-7811-4f7c-9f99-5431ac83b0fc_isrForCrosstalkSources_1031741_52.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1763ce61e7196451f13d56ce4b629a47 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/212cc9e2-c1c2-4318-9234-460446c55a54_isrForCrosstalkSources_1031739_3.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/212cc9e2-c1c2-4318-9234-460446c55a54_isrForCrosstalkSources_1031739_3.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b595c22df5fbbe8b349dd35db673a844 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcd5bc4a-4fa9-41f6-855a-cd72d9982533_isrForCrosstalkSources_1031741_49.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcd5bc4a-4fa9-41f6-855a-cd72d9982533_isrForCrosstalkSources_1031741_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:07 proc_lsst.multi:146 [INFO] found job 17153108 in provider astro proc_lsst.multi INFO: found job 17153108 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f10a4df54b5829c20b37c446b606f47 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7c09cc37-8b09-4fc5-8f10-eb9310ac4061_isrForCrosstalkSources_1031740_40.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7c09cc37-8b09-4fc5-8f10-eb9310ac4061_isrForCrosstalkSources_1031740_40.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa8c6dac326b90f348dc14dc6d65f8d with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d4f458b6-43c2-4b7f-85ea-a6e69887a257_isrForCrosstalkSources_1031741_53.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d4f458b6-43c2-4b7f-85ea-a6e69887a257_isrForCrosstalkSources_1031741_53.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1bf1d6986d70369513ff17ee0b3623b with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/95a9b941-0516-441b-9362-f0f1cc2a2789_isrForCrosstalkSources_1031740_37.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/95a9b941-0516-441b-9362-f0f1cc2a2789_isrForCrosstalkSources_1031740_37.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9d5d38d45e9f10fb49346fde7fcfd35 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4eb39acb-2d63-499c-8795-6d1e8f58bf92_isrForCrosstalkSources_1031741_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4eb39acb-2d63-499c-8795-6d1e8f58bf92_isrForCrosstalkSources_1031741_2.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d501444f063b425d9cc1c074bd6af745 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/34baec3d-1c34-4234-b7c5-55b176644ed2_isrForCrosstalkSources_1031740_55.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/34baec3d-1c34-4234-b7c5-55b176644ed2_isrForCrosstalkSources_1031740_55.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f042b25e0e4e0b649e394f4fdb23a37 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a0c600f-8705-4664-ac2f-3a9def889542_isrForCrosstalkSources_1031739_8.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a0c600f-8705-4664-ac2f-3a9def889542_isrForCrosstalkSources_1031739_8.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 000da5d445a851661624f3620e8503cb with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3363a383-4456-44a0-940f-97ffa601cda1_isrForCrosstalkSources_1031740_8.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/3363a383-4456-44a0-940f-97ffa601cda1_isrForCrosstalkSources_1031740_8.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcdd1d1b90ededf202c102fc754b6622 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/717e4d3a-b62b-407c-8a23-b75eee05b6b2_isrForCrosstalkSources_1031739_40.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/717e4d3a-b62b-407c-8a23-b75eee05b6b2_isrForCrosstalkSources_1031739_40.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ba6c7125e0eb97fadc09e36f6a7c087 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/28619702-0f8c-43f9-93b7-2490345d14d8_isrForCrosstalkSources_1031740_60.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/28619702-0f8c-43f9-93b7-2490345d14d8_isrForCrosstalkSources_1031740_60.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea6c74100fb8048ede6d10da82e15d99 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/db8e9bc2-1ccb-436e-9346-1ffd8140d904_isrForCrosstalkSources_1031740_47.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/db8e9bc2-1ccb-436e-9346-1ffd8140d904_isrForCrosstalkSources_1031740_47.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d1505fa9edb2d83c8532ba63852d22 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/13dac28a-c16d-418a-bc4d-2b1814ceeecc_isrForCrosstalkSources_1031740_50.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/13dac28a-c16d-418a-bc4d-2b1814ceeecc_isrForCrosstalkSources_1031740_50.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fda67ef2a43aa5f41fcc546e4372e1f1 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bab6ca0c-c448-4121-b11c-03afe26285f6_isrForCrosstalkSources_1031741_26.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bab6ca0c-c448-4121-b11c-03afe26285f6_isrForCrosstalkSources_1031741_26.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38334b4841e6669bbddcf44d6b30245b with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/2162ef67-5e9c-4f8b-bc28-41fc42de66d9_isrForCrosstalkSources_1031740_21.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/2162ef67-5e9c-4f8b-bc28-41fc42de66d9_isrForCrosstalkSources_1031740_21.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1dce9ab43f63dfd2f7b8276650e4182 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/04983e43-887b-407c-a5d3-d977d7c0854f_isrForCrosstalkSources_1031740_10.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/04983e43-887b-407c-a5d3-d977d7c0854f_isrForCrosstalkSources_1031740_10.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd763568fe041badf7f7d4f0fea04e1b with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7d71c92b-2fc6-4676-9e5d-b0a9ca076152_isrForCrosstalkSources_1031739_51.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/7d71c92b-2fc6-4676-9e5d-b0a9ca076152_isrForCrosstalkSources_1031739_51.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d2ed4817291bdf18fe5dc395053ebf3 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/42e590f9-b9be-4305-9443-fc30071040f6_isrForCrosstalkSources_1031739_30.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/42e590f9-b9be-4305-9443-fc30071040f6_isrForCrosstalkSources_1031739_30.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da396158757e4fb8ec62fd16384cc97c with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/97021850-6416-41fe-842c-7f3c1481290c_isrForCrosstalkSources_1031741_47.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/97021850-6416-41fe-842c-7f3c1481290c_isrForCrosstalkSources_1031741_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:14 proc_lsst.multi:146 [INFO] found job 17153110 in provider astro proc_lsst.multi INFO: found job 17153110 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2408a13faa91162dc7b7b202ae6012b with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e61e2d9f-f4e7-4516-b2ed-8f2a10963e52_isrForCrosstalkSources_1031739_61.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e61e2d9f-f4e7-4516-b2ed-8f2a10963e52_isrForCrosstalkSources_1031739_61.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c16208407d11e8d920b3866cd07c92f with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/baf1b6b3-524d-4bc7-ae87-4db7ccc740cd_isrForCrosstalkSources_1031741_60.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/baf1b6b3-524d-4bc7-ae87-4db7ccc740cd_isrForCrosstalkSources_1031741_60.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c9ada7b822297cba5bf60c5a5b69d69 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/927f969d-dd65-4d2b-ab79-224d3404d9d5_isrForCrosstalkSources_1031740_23.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/927f969d-dd65-4d2b-ab79-224d3404d9d5_isrForCrosstalkSources_1031740_23.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5ea9c44548dd7ea3712f84fa702840f with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/22a7e1cc-ea41-4bae-8651-820f3ae1ebde_isrForCrosstalkSources_1031739_47.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/22a7e1cc-ea41-4bae-8651-820f3ae1ebde_isrForCrosstalkSources_1031739_47.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6e4d1fce03a6ad47aae7f6450690d56 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0b8355ee-39db-4bcc-b982-5bba89b536e9_isrForCrosstalkSources_1031739_37.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0b8355ee-39db-4bcc-b982-5bba89b536e9_isrForCrosstalkSources_1031739_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:15 proc_lsst.multi:146 [INFO] found job 17153115 in provider astro proc_lsst.multi INFO: found job 17153115 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a321fa4043fe99978542baeb15defb94 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/246df2ce-466f-4a7a-a0ce-7fca17a7937b_isrForCrosstalkSources_1031739_60.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/246df2ce-466f-4a7a-a0ce-7fca17a7937b_isrForCrosstalkSources_1031739_60.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5919811922432f138295063317bc1a36 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a5dc967-da2e-40b8-9d37-5cf6f06fed83_isrForCrosstalkSources_1031740_17.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/0a5dc967-da2e-40b8-9d37-5cf6f06fed83_isrForCrosstalkSources_1031740_17.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295c080be7d61444d22cb6dec1b734d1 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/033c75a1-ff76-4983-9a36-154f13eacb32_isrForCrosstalkSources_1031741_54.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/033c75a1-ff76-4983-9a36-154f13eacb32_isrForCrosstalkSources_1031741_54.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02a6458209c0dd65a9e15707f5a450b4 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9efedbe4-9367-4404-97bd-46ff65d6a302_isrForCrosstalkSources_1031739_57.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9efedbe4-9367-4404-97bd-46ff65d6a302_isrForCrosstalkSources_1031739_57.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7556028dae25ccddc5356022abb5ee3b with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/275c803d-6cbb-4aee-9245-5344c4b64dc8_isrForCrosstalkSources_1031740_62.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/275c803d-6cbb-4aee-9245-5344c4b64dc8_isrForCrosstalkSources_1031740_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:16 proc_lsst.multi:146 [INFO] found job 17153117 in provider astro proc_lsst.multi INFO: found job 17153117 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d9d91c9ecb3f06be29be301d660b9a with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d2f4dae0-76f1-4e7a-90a7-0109a1a73747_isrForCrosstalkSources_1031739_29.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/d2f4dae0-76f1-4e7a-90a7-0109a1a73747_isrForCrosstalkSources_1031739_29.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d95ee633689aed6e3acbdcd2718750b with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/887b308b-d200-4ff9-8328-9eb3a95dd56c_isrForCrosstalkSources_1031740_26.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/887b308b-d200-4ff9-8328-9eb3a95dd56c_isrForCrosstalkSources_1031740_26.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c456121e0db42abfa323e0609673692 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/93a7d841-910e-4ba0-a2ef-0eeb4c3645c9_isrForCrosstalkSources_1031739_17.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/93a7d841-910e-4ba0-a2ef-0eeb4c3645c9_isrForCrosstalkSources_1031739_17.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0657319cb16b698e427056fb721d9316 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/3c079371-b0a5-4588-aab5-31aa6bedbcaf_isrForCrosstalkSources_1031741_46.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/3c079371-b0a5-4588-aab5-31aa6bedbcaf_isrForCrosstalkSources_1031741_46.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc935e0e632f43841e8726fc445bbdd6 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/71f9e55b-92b6-4372-bcec-f4b35ffea530_isrForCrosstalkSources_1031739_4.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/71f9e55b-92b6-4372-bcec-f4b35ffea530_isrForCrosstalkSources_1031739_4.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e1bcea6383d262c69a2288742583bf with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ec4838eb-bf55-41b6-8496-97c7cd8b090b_isrForCrosstalkSources_1031741_32.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ec4838eb-bf55-41b6-8496-97c7cd8b090b_isrForCrosstalkSources_1031741_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:17 proc_lsst.multi:146 [INFO] found job 17153118 in provider astro proc_lsst.multi INFO: found job 17153118 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d79543c7f3800a2b1fb7c1fc63c68af with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8e7be0da-3408-4bfc-bc0a-c94578075ce6_isrForCrosstalkSources_1031741_45.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/8e7be0da-3408-4bfc-bc0a-c94578075ce6_isrForCrosstalkSources_1031741_45.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 883e41641094f39b656080deb6a2702f with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/58bea049-75c7-49c7-ba75-ebd7f450e113_isrForCrosstalkSources_1031739_52.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/58bea049-75c7-49c7-ba75-ebd7f450e113_isrForCrosstalkSources_1031739_52.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 355f284b46e4c5b846c4f9c259af2276 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3c214be0-c7de-4d64-9b5e-58cc58855427_isrForCrosstalkSources_1031739_58.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/3c214be0-c7de-4d64-9b5e-58cc58855427_isrForCrosstalkSources_1031739_58.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ea0c389bda48b3bfb0dca73ca3ac72 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/feec20c6-8052-4576-bbf2-426bccb7aef8_isrForCrosstalkSources_1031739_15.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/feec20c6-8052-4576-bbf2-426bccb7aef8_isrForCrosstalkSources_1031739_15.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16038ff4d49957605a08510660db97d6 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/6e32e895-c634-49e5-83ad-7f0a67178c17_isrForCrosstalkSources_1031741_8.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/6e32e895-c634-49e5-83ad-7f0a67178c17_isrForCrosstalkSources_1031741_8.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4290e25727421889ba50c1416aab256d with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/523fd647-1618-48a9-a554-0be344b2878a_isrForCrosstalkSources_1031739_54.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/523fd647-1618-48a9-a554-0be344b2878a_isrForCrosstalkSources_1031739_54.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be8cb4a63c877bed059bf011aacd242d with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/da41a460-29ee-4702-81f2-0cdcd4db3298_isrForCrosstalkSources_1031741_37.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/da41a460-29ee-4702-81f2-0cdcd4db3298_isrForCrosstalkSources_1031741_37.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1882980b1b1edf47e126e3bc412e79be with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bfcc0c8e-7d16-431d-ad2d-e72bdee8f248_isrForCrosstalkSources_1031740_45.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/bfcc0c8e-7d16-431d-ad2d-e72bdee8f248_isrForCrosstalkSources_1031740_45.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 351692820d1caf3458c3c7d0ef1d7b42 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/44ce442e-f95f-43a5-8d17-16c3396a2bb1_isrForCrosstalkSources_1031739_62.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/44ce442e-f95f-43a5-8d17-16c3396a2bb1_isrForCrosstalkSources_1031739_62.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c85995e5567d46468184e37563caf27 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a987bc78-a60c-4ae9-9508-f2f4f7034409_isrForCrosstalkSources_1031741_39.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a987bc78-a60c-4ae9-9508-f2f4f7034409_isrForCrosstalkSources_1031741_39.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51c5761db455aee99756f5cf2df05fd5 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a51ce11a-b645-4d86-af84-183534281512_isrForCrosstalkSources_1031740_56.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a51ce11a-b645-4d86-af84-183534281512_isrForCrosstalkSources_1031740_56.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f710c1b0f0cc8037b8a7671aebae6391 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9e87fb25-9678-41eb-9e84-eba2004385e6_isrForCrosstalkSources_1031740_48.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/9e87fb25-9678-41eb-9e84-eba2004385e6_isrForCrosstalkSources_1031740_48.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c224f370c3cadb710d46504ca013a59c with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/98a9a5ec-5e81-43b9-b012-0993b786eb67_isrForCrosstalkSources_1031741_57.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/98a9a5ec-5e81-43b9-b012-0993b786eb67_isrForCrosstalkSources_1031741_57.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55659f55cb0031e1c3b8b6b452fed010 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/cee77076-7350-420c-ba5a-03640c79fcda_isrForCrosstalkSources_1031740_44.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/cee77076-7350-420c-ba5a-03640c79fcda_isrForCrosstalkSources_1031740_44.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36e4293f685552e3bb36c12a8b242761 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4ad3deac-751f-4837-8093-b15fdfa857ab_isrForCrosstalkSources_1031741_6.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/4ad3deac-751f-4837-8093-b15fdfa857ab_isrForCrosstalkSources_1031741_6.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87a599071d15a3b5aee33c89498ad7c9 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/26415ca8-b126-4dba-9c60-adbe9933619a_isrForCrosstalkSources_1031739_6.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/26415ca8-b126-4dba-9c60-adbe9933619a_isrForCrosstalkSources_1031739_6.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7140adb7f499b9e60763dd0564b96f1d with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7fbcac33-8831-495a-8367-791ccfbe85ff_isrForCrosstalkSources_1031740_2.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7fbcac33-8831-495a-8367-791ccfbe85ff_isrForCrosstalkSources_1031740_2.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ea660bd77d705938718a8a3f50d4d2 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/85b9ea55-c7ff-443c-bcb6-858c32e76eec_isrForCrosstalkSources_1031741_62.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/85b9ea55-c7ff-443c-bcb6-858c32e76eec_isrForCrosstalkSources_1031741_62.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2727561c4be1cf066dce1118b51a0ba2 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ecf40eec-8669-48c4-a129-ace4dc33e91c_isrForCrosstalkSources_1031741_36.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/ecf40eec-8669-48c4-a129-ace4dc33e91c_isrForCrosstalkSources_1031741_36.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ebf9c74bfc507ab1d1dc167b714814 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/70e79934-0aba-43bf-80c0-7317e264b548_isrForCrosstalkSources_1031739_26.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/70e79934-0aba-43bf-80c0-7317e264b548_isrForCrosstalkSources_1031739_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:18 proc_lsst.multi:146 [INFO] found job 17153119 in provider astro proc_lsst.multi INFO: found job 17153119 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:18 proc_lsst.multi:146 [INFO] found job 17153120 in provider astro proc_lsst.multi INFO: found job 17153120 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c577fbfee86641b4b3fd83e291b79503 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b6f3cca8-1a71-42d9-b322-344a31fd0078_isrForCrosstalkSources_1031741_21.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b6f3cca8-1a71-42d9-b322-344a31fd0078_isrForCrosstalkSources_1031741_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:19 proc_lsst.multi:146 [INFO] found job 17153121 in provider astro proc_lsst.multi INFO: found job 17153121 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:20 proc_lsst.multi:146 [INFO] found job 17153123 in provider astro proc_lsst.multi INFO: found job 17153123 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:20 proc_lsst.multi:146 [INFO] found job 17153125 in provider astro proc_lsst.multi INFO: found job 17153125 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:21 proc_lsst.multi:146 [INFO] found job 17153127 in provider astro proc_lsst.multi INFO: found job 17153127 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c66379569d20e67643fb498a60584268 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/231e7c16-9a6a-49ad-9592-df5362379c1b_isrForCrosstalkSources_1031741_3.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/231e7c16-9a6a-49ad-9592-df5362379c1b_isrForCrosstalkSources_1031741_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:22 proc_lsst.multi:146 [INFO] found job 17153129 in provider astro proc_lsst.multi INFO: found job 17153129 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:22 proc_lsst.multi:146 [INFO] found job 17153135 in provider astro proc_lsst.multi INFO: found job 17153135 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d4174f74f1ce58a4da51ceda4f8aa15 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/625561d9-199c-483c-a508-21b9b79775a7_isrForCrosstalkSources_1031740_6.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/625561d9-199c-483c-a508-21b9b79775a7_isrForCrosstalkSources_1031740_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:23 proc_lsst.multi:146 [INFO] found job 17153137 in provider astro proc_lsst.multi INFO: found job 17153137 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d59a7a571e4f97fd7eac34dc7f148dc with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cdd9fb65-640f-4675-bcd9-c6c729b801c8_isrForCrosstalkSources_1031741_18.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cdd9fb65-640f-4675-bcd9-c6c729b801c8_isrForCrosstalkSources_1031741_18.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 255eb383ced93abe1a349a6f34e471fd with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/acfbb67c-9dce-4c8b-8cef-b52fc3cc3d2c_isrForCrosstalkSources_1031741_38.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/acfbb67c-9dce-4c8b-8cef-b52fc3cc3d2c_isrForCrosstalkSources_1031741_38.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02c147d47c6fc84e03176c4a0f41501 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b64c3e17-3759-4fe9-8041-9bd23c08d83a_isrForCrosstalkSources_1031741_58.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/b64c3e17-3759-4fe9-8041-9bd23c08d83a_isrForCrosstalkSources_1031741_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:23 proc_lsst.multi:146 [INFO] found job 17153140 in provider astro proc_lsst.multi INFO: found job 17153140 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153088,17153089,17153090,17153091,17153093,17153094,17153096,17153097,17153098,17153099,17153100,17153101,17153103,17153104,17153106,17153108,17153110,17153115,17153117,17153118,17153119,17153120,17153121,17153123,17153125,17153127,17153129,17153135,17153137,17153140' parsl.providers.slurm.slurm DEBUG: squeue returned 17153140 R 17153135 R 17153137 R 17153127 R 17153129 R 17153123 R 17153125 R 17153120 R 17153121 R 17153119 R 17153110 R 17153115 R 17153117 R 17153118 R 17153103 R 17153104 R 17153106 R 17153108 R 17153098 R 17153099 R 17153100 R 17153101 R 17153097 R 17153096 R 17153093 R 17153094 R 17153088 R 17153089 R 17153090 R 17153091 R parsl.providers.slurm.slurm DEBUG: Updating job 17153140 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153135 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153137 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153127 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153129 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153123 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153125 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153120 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153121 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153119 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153110 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153115 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153117 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153118 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153103 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153104 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153106 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153108 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153098 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153099 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153100 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153101 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153097 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153096 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153093 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153094 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153088 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153089 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153090 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153091 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:24 proc_lsst.multi:146 [INFO] found job 17153142 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153142 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153142,17153145,17153146,17153149,17153151' parsl.providers.slurm.slurm DEBUG: squeue returned 17153149 R 17153151 R 17153146 R 17153145 R 17153142 R parsl.providers.slurm.slurm DEBUG: Updating job 17153149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153142 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:25 proc_lsst.multi:146 [INFO] found job 17153145 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153145 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153142,17153145,17153146,17153149,17153151' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bedb776a6035a5f5e40fd941cf26ac02 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/057526d1-9792-4f59-84ad-1276b0b73ae9_isrForCrosstalkSources_1031741_17.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/057526d1-9792-4f59-84ad-1276b0b73ae9_isrForCrosstalkSources_1031741_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153149 R 17153151 R 17153146 R 17153145 R 17153142 R parsl.providers.slurm.slurm DEBUG: Updating job 17153149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153142 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:25 proc_lsst.multi:146 [INFO] found job 17153146 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153146 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153142,17153145,17153146,17153149,17153151' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de0d397d7494b9db33395c5d194f44df with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5ca6369d-6cae-4f3e-9374-165ad857b986_isrForCrosstalkSources_1031741_34.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/5ca6369d-6cae-4f3e-9374-165ad857b986_isrForCrosstalkSources_1031741_34.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43665893e8f50e1264ce1fcdddbba0c8 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/de53b6f1-e05b-4196-916c-ec4575991ab2_isrForCrosstalkSources_1031741_20.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/de53b6f1-e05b-4196-916c-ec4575991ab2_isrForCrosstalkSources_1031741_20.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd795448dc45f467d7696fc48c9318cd with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bf6a07a1-1364-42c2-b871-60095813774e_isrForCrosstalkSources_1031741_24.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/bf6a07a1-1364-42c2-b871-60095813774e_isrForCrosstalkSources_1031741_24.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d28685d92f2f1bb25a10706784e0c469 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aebe1a92-c13a-48e8-a0b4-c271ffedb42b_isrForCrosstalkSources_1031739_34.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aebe1a92-c13a-48e8-a0b4-c271ffedb42b_isrForCrosstalkSources_1031739_34.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 744220a376ee8e305983eae876488d7c with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/a3d3136c-4886-4e3c-b36a-0e58f9f9c572_isrForCrosstalkSources_1031739_33.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/a3d3136c-4886-4e3c-b36a-0e58f9f9c572_isrForCrosstalkSources_1031739_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153149 R 17153151 R 17153146 R 17153145 R 17153142 R parsl.providers.slurm.slurm DEBUG: Updating job 17153149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153142 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:26 proc_lsst.multi:146 [INFO] found job 17153149 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153149 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153142,17153145,17153146,17153149,17153151' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52718b7cf4f3d290ef5c482ed5635cd4 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ac78428-4e49-40ae-817b-d0d8aca51601_isrForCrosstalkSources_1031740_29.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/8ac78428-4e49-40ae-817b-d0d8aca51601_isrForCrosstalkSources_1031740_29.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0bbdae80e62d8847c9db4a0d1f6ee3c with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/f6eefef9-cf01-433e-8609-ed0c0c644788_isrForCrosstalkSources_1031741_23.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/f6eefef9-cf01-433e-8609-ed0c0c644788_isrForCrosstalkSources_1031741_23.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d2f0fdacc845e8a8eae6fa84c06946 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/6e0eea99-eb7b-4d11-82b5-c1c8c8764511_isrForCrosstalkSources_1031739_27.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/6e0eea99-eb7b-4d11-82b5-c1c8c8764511_isrForCrosstalkSources_1031739_27.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef689de3f9197f9831b487808f47ffd5 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/24b4118b-539c-45b9-bf68-741dcf0697e2_isrForCrosstalkSources_1031740_5.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/24b4118b-539c-45b9-bf68-741dcf0697e2_isrForCrosstalkSources_1031740_5.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7008824c6b8dce5d47a6b9c5a70af84b with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0485b3bc-9d74-4e31-bd0d-a6c5fbad03f9_isrForCrosstalkSources_1031741_14.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/0485b3bc-9d74-4e31-bd0d-a6c5fbad03f9_isrForCrosstalkSources_1031741_14.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d70733011c0285c742bca386cc7f851 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aa64a890-ec0e-4552-91c2-a14c9db406d2_isrForCrosstalkSources_1031739_28.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/aa64a890-ec0e-4552-91c2-a14c9db406d2_isrForCrosstalkSources_1031739_28.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2956e592eeba46f2def1edbf9bd18aff with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0cfbf8a6-15c5-465a-8341-f96cef7dd4bb_isrForCrosstalkSources_1031739_31.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/0cfbf8a6-15c5-465a-8341-f96cef7dd4bb_isrForCrosstalkSources_1031739_31.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd54ebcd4658e34b9cb9f072e6426b3 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/95d7a0c3-3d3a-4c4d-980c-6ebaeb101f27_isrForCrosstalkSources_1031739_48.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/95d7a0c3-3d3a-4c4d-980c-6ebaeb101f27_isrForCrosstalkSources_1031739_48.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3a48999cec90ad47b292bf15cc60d3c with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a8da8dff-c3c5-4a87-83ab-ec5ea23e591f_isrForCrosstalkSources_1031741_15.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a8da8dff-c3c5-4a87-83ab-ec5ea23e591f_isrForCrosstalkSources_1031741_15.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 249b8acf0e2809b579ba0fcf58d5e51a with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/600198e2-33e3-4363-88de-cb291d13b25a_isrForCrosstalkSources_1031740_34.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/600198e2-33e3-4363-88de-cb291d13b25a_isrForCrosstalkSources_1031740_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153149 R 17153151 R 17153146 R 17153145 R 17153142 R parsl.providers.slurm.slurm DEBUG: Updating job 17153149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153142 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:26 proc_lsst.multi:146 [INFO] found job 17153151 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153151 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153142,17153145,17153146,17153149,17153151' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09830bc5ebdd0c6de82a4430d9b8691c with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7577a54c-25b0-432d-a4c0-8de016e5be09_isrForCrosstalkSources_1031740_59.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/7577a54c-25b0-432d-a4c0-8de016e5be09_isrForCrosstalkSources_1031740_59.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c64337766f127a353b3616b3fd7f981 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/69a2c0be-235d-49d9-8a18-dd977e83cd17_isrForCrosstalkSources_1031740_42.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/69a2c0be-235d-49d9-8a18-dd977e83cd17_isrForCrosstalkSources_1031740_42.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc25155371fff8384abdd4c4de6f39d with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/97630da0-37c1-4041-ada7-9984360fe5c2_isrForCrosstalkSources_1031739_7.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/97630da0-37c1-4041-ada7-9984360fe5c2_isrForCrosstalkSources_1031739_7.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70316cb6f90f91df5522549dfdbc3849 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/09bc09b1-8f5c-46bf-aeb2-6dd6de3bf1f4_isrForCrosstalkSources_1031739_32.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/09bc09b1-8f5c-46bf-aeb2-6dd6de3bf1f4_isrForCrosstalkSources_1031739_32.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bccfc2bfadb0021d291e8562bc29933 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/99378acb-95e5-4fd8-b5bb-9b555dcaecfb_isrForCrosstalkSources_1031740_53.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/99378acb-95e5-4fd8-b5bb-9b555dcaecfb_isrForCrosstalkSources_1031740_53.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b8a458bea0965e7b848e82b05bbf3a with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cbb811e4-be87-4073-bbcb-a613325593ec_isrForCrosstalkSources_1031741_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/cbb811e4-be87-4073-bbcb-a613325593ec_isrForCrosstalkSources_1031741_55.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry defe9f117c2cafd0168b5ed655b5e439 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d9f3371-b706-4bb3-a28b-1739445a084e_isrForCrosstalkSources_1031739_39.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/1d9f3371-b706-4bb3-a28b-1739445a084e_isrForCrosstalkSources_1031739_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153149 R 17153151 R 17153146 R 17153145 R 17153142 R parsl.providers.slurm.slurm DEBUG: Updating job 17153149 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153151 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153146 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153145 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153142 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:26 proc_lsst.multi:146 [INFO] found job 17153153 in provider escience_a40 proc_lsst.multi INFO: found job 17153153 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3498fdcd6c81885b4d1a4ff93440efa with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a39552b6-53d3-41c8-bfa7-f13de5b27572_isrForCrosstalkSources_1031741_50.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/a39552b6-53d3-41c8-bfa7-f13de5b27572_isrForCrosstalkSources_1031741_50.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53bef407340a7ae7d0b4c8a87fe797ab with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9bab137d-50cc-4028-b5f0-ea6bf40db059_isrForCrosstalkSources_1031741_30.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/9bab137d-50cc-4028-b5f0-ea6bf40db059_isrForCrosstalkSources_1031741_30.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b38aaeea92efe67e69c5dd127250c30 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c478d5b2-e757-4d99-b571-efab00dbdce7_isrForCrosstalkSources_1031740_12.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/c478d5b2-e757-4d99-b571-efab00dbdce7_isrForCrosstalkSources_1031740_12.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03927ac4183536c93e25d2350b3516bc with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/76263766-8721-4510-ae31-8828fb5f3723_isrForCrosstalkSources_1031740_41.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/76263766-8721-4510-ae31-8828fb5f3723_isrForCrosstalkSources_1031740_41.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e14d12ae54b46481e2872ebb1dcb1c8 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4bc2bebf-1823-4431-9266-c83692fd4e24_isrForCrosstalkSources_1031739_46.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/4bc2bebf-1823-4431-9266-c83692fd4e24_isrForCrosstalkSources_1031739_46.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c72d2582d39333f38157066c28d4fbc9 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a2636c8d-2797-4d13-a63b-c841f9567100_isrForCrosstalkSources_1031740_16.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a2636c8d-2797-4d13-a63b-c841f9567100_isrForCrosstalkSources_1031740_16.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3add3798ebd76020d1102f45640062b8 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a3cd226-1d91-41e6-b08e-1c78f0a0286c_isrForCrosstalkSources_1031739_25.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/9a3cd226-1d91-41e6-b08e-1c78f0a0286c_isrForCrosstalkSources_1031739_25.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ded85bcb67685c4e7d1fd4b9bb6e0b with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e40b2ad9-01ee-4289-94f4-38bc18011548_isrForCrosstalkSources_1031739_24.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/e40b2ad9-01ee-4289-94f4-38bc18011548_isrForCrosstalkSources_1031739_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:27 proc_lsst.multi:146 [INFO] found job 17153155 in provider escience_a40 proc_lsst.multi INFO: found job 17153155 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dcaf7fb1791e18c85a7c5735dc3ecf9 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/32d97d2a-71c5-4d3b-9304-a41d922e3c0d_isrForCrosstalkSources_1031741_19.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/32d97d2a-71c5-4d3b-9304-a41d922e3c0d_isrForCrosstalkSources_1031741_19.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f217298404c3e850c6337ba5a4eac539 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/99fbf786-5be6-4f19-95fa-5e0385bd9306_isrForCrosstalkSources_1031739_20.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/99fbf786-5be6-4f19-95fa-5e0385bd9306_isrForCrosstalkSources_1031739_20.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7e07062ecb7f5db9d5190169a097ca8 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2a4202aa-77a9-47fd-a166-7a18fecc5de7_isrForCrosstalkSources_1031741_4.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/2a4202aa-77a9-47fd-a166-7a18fecc5de7_isrForCrosstalkSources_1031741_4.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebc3ea3eaf9c4346bd61de0ea929a0be with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/35e2256c-22d2-4a01-89f8-f5bcaa4875d8_isrForCrosstalkSources_1031741_44.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/35e2256c-22d2-4a01-89f8-f5bcaa4875d8_isrForCrosstalkSources_1031741_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:27 proc_lsst.multi:146 [INFO] found job 17153157 in provider escience_a40 proc_lsst.multi INFO: found job 17153157 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 320773265bdbcfd2c28ae3b34fbc3d5b with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/1bacc129-4bb0-4c11-b52d-09d1f3b03b31_isrForCrosstalkSources_1031740_61.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/1bacc129-4bb0-4c11-b52d-09d1f3b03b31_isrForCrosstalkSources_1031740_61.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ac1c6ed3bfa6ed11ab3b2a09fd5af2 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a181f316-17bc-4a00-8e33-c1b4d6e05ba6_isrForCrosstalkSources_1031740_52.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/a181f316-17bc-4a00-8e33-c1b4d6e05ba6_isrForCrosstalkSources_1031740_52.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cc8b2bf4f04a5dbc55c4c86db65916b with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/cc8d7399-8f84-4b90-81b4-5d109e890b2b_isrForCrosstalkSources_1031739_50.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031739/cc8d7399-8f84-4b90-81b4-5d109e890b2b_isrForCrosstalkSources_1031739_50.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c7f39f5d73d7d7734d7c47c1e9a6565 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcaf6bb1-c3b8-40e4-84ad-557e9a094286_isrForCrosstalkSources_1031741_13.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/dcaf6bb1-c3b8-40e4-84ad-557e9a094286_isrForCrosstalkSources_1031741_13.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3ae27b61a09975fa0fc422e76190101 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d1bd05e8-7df8-40c0-85c7-b5d0c0d617b3_isrForCrosstalkSources_1031741_7.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031741/d1bd05e8-7df8-40c0-85c7-b5d0c0d617b3_isrForCrosstalkSources_1031741_7.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c44b3b17700688a821de9efaaaaf82e7 with result from task 177 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b7113960-cb4f-4b57-a64b-601623ca22fd_isrForCrosstalkSources_1031740_35.stdout parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/submit/DEEP/20210909/B1b/science#step0/20240317T070413Z/logs/isrForCrosstalkSources/1031740/b7113960-cb4f-4b57-a64b-601623ca22fd_isrForCrosstalkSources_1031740_35.stderr parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:28 proc_lsst.multi:146 [INFO] found job 17153159 in provider escience_a40 proc_lsst.multi INFO: found job 17153159 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:28 proc_lsst.multi:146 [INFO] found job 17153161 in provider escience_a40 proc_lsst.multi INFO: found job 17153161 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:28 proc_lsst.multi:146 [INFO] found job 17153163 in provider escience_a40 proc_lsst.multi INFO: found job 17153163 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:28 proc_lsst.multi:146 [INFO] found job 17153165 in provider escience_a40 proc_lsst.multi INFO: found job 17153165 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:29 proc_lsst.multi:146 [INFO] found job 17153167 in provider escience_a40 proc_lsst.multi INFO: found job 17153167 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:29 proc_lsst.multi:146 [INFO] found job 17153169 in provider escience_a40 proc_lsst.multi INFO: found job 17153169 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:29 proc_lsst.multi:146 [INFO] found job 17153171 in provider escience_a40 proc_lsst.multi INFO: found job 17153171 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING 2024-03-17 07:06:29 proc_lsst.multi:146 [INFO] found job 17153173 in provider escience_a40 proc_lsst.multi INFO: found job 17153173 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17153153,17153155,17153157,17153159,17153161,17153163,17153165,17153167,17153169,17153171,17153173' parsl.providers.slurm.slurm DEBUG: squeue returned 17153173 R 17153167 R 17153169 R 17153171 R 17153161 R 17153163 R 17153165 R 17153157 R 17153159 R 17153155 R 17153153 R parsl.providers.slurm.slurm DEBUG: Updating job 17153173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153161 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153157 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153159 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153155 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17153153 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 47, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 47/0 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (47) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 9.5367431640625e-07s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22732096912592 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=47, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 47 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ca3e131ada1b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ca3e131ada1b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 90bcff44c5eb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 90bcff44c5eb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9cc06fb3d360 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9cc06fb3d360 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 95dc2406ad1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 95dc2406ad1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e23b5b81b4df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e23b5b81b4df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1c3fd27c03a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1c3fd27c03a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9f1e9487fb7b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9f1e9487fb7b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0349ff28610d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0349ff28610d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 553bb1a514af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 553bb1a514af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 24ff2c924b6a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 24ff2c924b6a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 59a051a8fc9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 59a051a8fc9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dbd263f5d196 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dbd263f5d196 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66fc491888a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66fc491888a0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81f238ba68ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81f238ba68ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 072596dd6c48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 072596dd6c48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d00581dd038 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d00581dd038 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d05385b1b38 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d05385b1b38 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb598f8cded5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb598f8cded5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29c76e1b5422 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29c76e1b5422 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2c20ff527a4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2c20ff527a4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94364b79c96d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94364b79c96d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 528950893ccf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 528950893ccf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8b8a47a23aa8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8b8a47a23aa8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 12e5251ad13b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 12e5251ad13b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7ee901f16d3b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7ee901f16d3b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 53077abc3dfe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 53077abc3dfe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1eab8571a48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1eab8571a48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d17bc561788 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d17bc561788 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 15b222265f89 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 15b222265f89 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4c263c880ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4c263c880ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b10a7c920be2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b10a7c920be2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c59b3f7d022c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c59b3f7d022c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33cd616e8bac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33cd616e8bac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f39031d247eb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f39031d247eb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64f947ce697a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64f947ce697a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 59d6cb311183 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 59d6cb311183 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f11efbebb26 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f11efbebb26 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5079f0229762 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5079f0229762 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 219a7e460230 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 219a7e460230 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5857fdfd70fa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5857fdfd70fa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bfdf8e696586 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bfdf8e696586 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a5c15addcd6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a5c15addcd6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ef530a2431f3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ef530a2431f3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4881100ad904 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4881100ad904 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff28b51bed2a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff28b51bed2a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6800817baf7f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6800817baf7f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ede46554ee19 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ede46554ee19 2024-03-17 07:06:30 proc_lsst.multi:146 [INFO] found job 17153096 in provider astro proc_lsst.multi INFO: found job 17153096 in provider astro 2024-03-17 07:06:30 proc_lsst.multi:201 [INFO] cancelling 17153096 on provider astro proc_lsst.multi INFO: cancelling 17153096 on provider astro 2024-03-17 07:06:30 proc_lsst.multi:146 [INFO] found job 17153094 in provider astro proc_lsst.multi INFO: found job 17153094 in provider astro 2024-03-17 07:06:30 proc_lsst.multi:201 [INFO] cancelling 17153094 on provider astro proc_lsst.multi INFO: cancelling 17153094 on provider astro 2024-03-17 07:06:30 proc_lsst.multi:146 [INFO] found job 17153169 in provider escience_a40 proc_lsst.multi INFO: found job 17153169 in provider escience_a40 2024-03-17 07:06:30 proc_lsst.multi:201 [INFO] cancelling 17153169 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153169 on provider escience_a40 2024-03-17 07:06:30 proc_lsst.multi:146 [INFO] found job 17153097 in provider astro proc_lsst.multi INFO: found job 17153097 in provider astro 2024-03-17 07:06:30 proc_lsst.multi:201 [INFO] cancelling 17153097 on provider astro proc_lsst.multi INFO: cancelling 17153097 on provider astro 2024-03-17 07:06:31 proc_lsst.multi:146 [INFO] found job 17153159 in provider escience_a40 proc_lsst.multi INFO: found job 17153159 in provider escience_a40 2024-03-17 07:06:31 proc_lsst.multi:201 [INFO] cancelling 17153159 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153159 on provider escience_a40 2024-03-17 07:06:31 proc_lsst.multi:146 [INFO] found job 17153157 in provider escience_a40 proc_lsst.multi INFO: found job 17153157 in provider escience_a40 2024-03-17 07:06:31 proc_lsst.multi:201 [INFO] cancelling 17153157 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153157 on provider escience_a40 2024-03-17 07:06:31 proc_lsst.multi:146 [INFO] found job 17153118 in provider astro proc_lsst.multi INFO: found job 17153118 in provider astro 2024-03-17 07:06:31 proc_lsst.multi:201 [INFO] cancelling 17153118 on provider astro proc_lsst.multi INFO: cancelling 17153118 on provider astro 2024-03-17 07:06:31 proc_lsst.multi:146 [INFO] found job 53768 in provider local proc_lsst.multi INFO: found job 53768 in provider local 2024-03-17 07:06:31 proc_lsst.multi:201 [INFO] cancelling 53768 on provider local proc_lsst.multi INFO: cancelling 53768 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53768 2024-03-17 07:06:32 proc_lsst.multi:146 [INFO] found job 17153171 in provider escience_a40 proc_lsst.multi INFO: found job 17153171 in provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:201 [INFO] cancelling 17153171 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153171 on provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:146 [INFO] found job 17153173 in provider escience_a40 proc_lsst.multi INFO: found job 17153173 in provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:201 [INFO] cancelling 17153173 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153173 on provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:146 [INFO] found job 17153115 in provider astro proc_lsst.multi INFO: found job 17153115 in provider astro 2024-03-17 07:06:32 proc_lsst.multi:201 [INFO] cancelling 17153115 on provider astro proc_lsst.multi INFO: cancelling 17153115 on provider astro 2024-03-17 07:06:32 proc_lsst.multi:146 [INFO] found job 17153155 in provider escience_a40 proc_lsst.multi INFO: found job 17153155 in provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:201 [INFO] cancelling 17153155 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153155 on provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:146 [INFO] found job 17153167 in provider escience_a40 proc_lsst.multi INFO: found job 17153167 in provider escience_a40 2024-03-17 07:06:32 proc_lsst.multi:201 [INFO] cancelling 17153167 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153167 on provider escience_a40 2024-03-17 07:06:33 proc_lsst.multi:146 [INFO] found job 17153153 in provider escience_a40 proc_lsst.multi INFO: found job 17153153 in provider escience_a40 2024-03-17 07:06:33 proc_lsst.multi:201 [INFO] cancelling 17153153 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153153 on provider escience_a40 2024-03-17 07:06:33 proc_lsst.multi:146 [INFO] found job 17153119 in provider astro proc_lsst.multi INFO: found job 17153119 in provider astro 2024-03-17 07:06:33 proc_lsst.multi:201 [INFO] cancelling 17153119 on provider astro proc_lsst.multi INFO: cancelling 17153119 on provider astro 2024-03-17 07:06:33 proc_lsst.multi:146 [INFO] found job 17153163 in provider escience_a40 proc_lsst.multi INFO: found job 17153163 in provider escience_a40 2024-03-17 07:06:33 proc_lsst.multi:201 [INFO] cancelling 17153163 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153163 on provider escience_a40 2024-03-17 07:06:33 proc_lsst.multi:146 [INFO] found job 17153165 in provider escience_a40 proc_lsst.multi INFO: found job 17153165 in provider escience_a40 2024-03-17 07:06:33 proc_lsst.multi:201 [INFO] cancelling 17153165 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153165 on provider escience_a40 2024-03-17 07:06:34 proc_lsst.multi:146 [INFO] found job 17153161 in provider escience_a40 proc_lsst.multi INFO: found job 17153161 in provider escience_a40 2024-03-17 07:06:34 proc_lsst.multi:201 [INFO] cancelling 17153161 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153161 on provider escience_a40 2024-03-17 07:06:34 proc_lsst.multi:146 [INFO] found job 17153137 in provider astro proc_lsst.multi INFO: found job 17153137 in provider astro 2024-03-17 07:06:34 proc_lsst.multi:201 [INFO] cancelling 17153137 on provider astro proc_lsst.multi INFO: cancelling 17153137 on provider astro 2024-03-17 07:06:34 proc_lsst.multi:146 [INFO] found job 17153117 in provider astro proc_lsst.multi INFO: found job 17153117 in provider astro 2024-03-17 07:06:34 proc_lsst.multi:201 [INFO] cancelling 17153117 on provider astro proc_lsst.multi INFO: cancelling 17153117 on provider astro 2024-03-17 07:06:34 proc_lsst.multi:146 [INFO] found job 17153140 in provider astro proc_lsst.multi INFO: found job 17153140 in provider astro 2024-03-17 07:06:34 proc_lsst.multi:201 [INFO] cancelling 17153140 on provider astro proc_lsst.multi INFO: cancelling 17153140 on provider astro 2024-03-17 07:06:34 proc_lsst.multi:146 [INFO] found job 17153127 in provider astro proc_lsst.multi INFO: found job 17153127 in provider astro 2024-03-17 07:06:34 proc_lsst.multi:201 [INFO] cancelling 17153127 on provider astro proc_lsst.multi INFO: cancelling 17153127 on provider astro 2024-03-17 07:06:35 proc_lsst.multi:146 [INFO] found job 17153135 in provider astro proc_lsst.multi INFO: found job 17153135 in provider astro 2024-03-17 07:06:35 proc_lsst.multi:201 [INFO] cancelling 17153135 on provider astro proc_lsst.multi INFO: cancelling 17153135 on provider astro 2024-03-17 07:06:35 proc_lsst.multi:146 [INFO] found job 17153129 in provider astro proc_lsst.multi INFO: found job 17153129 in provider astro 2024-03-17 07:06:35 proc_lsst.multi:201 [INFO] cancelling 17153129 on provider astro proc_lsst.multi INFO: cancelling 17153129 on provider astro 2024-03-17 07:06:35 proc_lsst.multi:146 [INFO] found job 17153101 in provider astro proc_lsst.multi INFO: found job 17153101 in provider astro 2024-03-17 07:06:35 proc_lsst.multi:201 [INFO] cancelling 17153101 on provider astro proc_lsst.multi INFO: cancelling 17153101 on provider astro 2024-03-17 07:06:35 proc_lsst.multi:146 [INFO] found job 17153149 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153149 in provider escience_rtx6k 2024-03-17 07:06:35 proc_lsst.multi:201 [INFO] cancelling 17153149 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153149 on provider escience_rtx6k 2024-03-17 07:06:35 proc_lsst.multi:146 [INFO] found job 17153146 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153146 in provider escience_rtx6k 2024-03-17 07:06:35 proc_lsst.multi:201 [INFO] cancelling 17153146 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153146 on provider escience_rtx6k 2024-03-17 07:06:36 proc_lsst.multi:146 [INFO] found job 17153089 in provider astro proc_lsst.multi INFO: found job 17153089 in provider astro 2024-03-17 07:06:36 proc_lsst.multi:201 [INFO] cancelling 17153089 on provider astro proc_lsst.multi INFO: cancelling 17153089 on provider astro 2024-03-17 07:06:36 proc_lsst.multi:146 [INFO] found job 17153106 in provider astro proc_lsst.multi INFO: found job 17153106 in provider astro 2024-03-17 07:06:36 proc_lsst.multi:201 [INFO] cancelling 17153106 on provider astro proc_lsst.multi INFO: cancelling 17153106 on provider astro 2024-03-17 07:06:36 proc_lsst.multi:146 [INFO] found job 17153100 in provider astro proc_lsst.multi INFO: found job 17153100 in provider astro 2024-03-17 07:06:36 proc_lsst.multi:201 [INFO] cancelling 17153100 on provider astro proc_lsst.multi INFO: cancelling 17153100 on provider astro 2024-03-17 07:06:36 proc_lsst.multi:146 [INFO] found job 17153142 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153142 in provider escience_rtx6k 2024-03-17 07:06:36 proc_lsst.multi:201 [INFO] cancelling 17153142 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153142 on provider escience_rtx6k 2024-03-17 07:06:37 proc_lsst.multi:146 [INFO] found job 17153090 in provider astro proc_lsst.multi INFO: found job 17153090 in provider astro 2024-03-17 07:06:37 proc_lsst.multi:201 [INFO] cancelling 17153090 on provider astro proc_lsst.multi INFO: cancelling 17153090 on provider astro 2024-03-17 07:06:37 proc_lsst.multi:146 [INFO] found job 17153093 in provider astro proc_lsst.multi INFO: found job 17153093 in provider astro 2024-03-17 07:06:37 proc_lsst.multi:201 [INFO] cancelling 17153093 on provider astro proc_lsst.multi INFO: cancelling 17153093 on provider astro 2024-03-17 07:06:37 proc_lsst.multi:146 [INFO] found job 17153091 in provider astro proc_lsst.multi INFO: found job 17153091 in provider astro 2024-03-17 07:06:37 proc_lsst.multi:201 [INFO] cancelling 17153091 on provider astro proc_lsst.multi INFO: cancelling 17153091 on provider astro 2024-03-17 07:06:37 proc_lsst.multi:146 [INFO] found job 17153110 in provider astro proc_lsst.multi INFO: found job 17153110 in provider astro 2024-03-17 07:06:37 proc_lsst.multi:201 [INFO] cancelling 17153110 on provider astro proc_lsst.multi INFO: cancelling 17153110 on provider astro 2024-03-17 07:06:38 proc_lsst.multi:146 [INFO] found job 17153145 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153145 in provider escience_rtx6k 2024-03-17 07:06:38 proc_lsst.multi:201 [INFO] cancelling 17153145 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153145 on provider escience_rtx6k 2024-03-17 07:06:38 proc_lsst.multi:146 [INFO] found job 17153104 in provider astro proc_lsst.multi INFO: found job 17153104 in provider astro 2024-03-17 07:06:38 proc_lsst.multi:201 [INFO] cancelling 17153104 on provider astro proc_lsst.multi INFO: cancelling 17153104 on provider astro 2024-03-17 07:06:38 proc_lsst.multi:146 [INFO] found job 17153099 in provider astro proc_lsst.multi INFO: found job 17153099 in provider astro 2024-03-17 07:06:38 proc_lsst.multi:201 [INFO] cancelling 17153099 on provider astro proc_lsst.multi INFO: cancelling 17153099 on provider astro 2024-03-17 07:06:39 proc_lsst.multi:146 [INFO] found job 17153108 in provider astro proc_lsst.multi INFO: found job 17153108 in provider astro 2024-03-17 07:06:39 proc_lsst.multi:201 [INFO] cancelling 17153108 on provider astro proc_lsst.multi INFO: cancelling 17153108 on provider astro 2024-03-17 07:06:39 proc_lsst.multi:146 [INFO] found job 17153103 in provider astro proc_lsst.multi INFO: found job 17153103 in provider astro 2024-03-17 07:06:39 proc_lsst.multi:201 [INFO] cancelling 17153103 on provider astro proc_lsst.multi INFO: cancelling 17153103 on provider astro 2024-03-17 07:06:39 proc_lsst.multi:146 [INFO] found job 17153088 in provider astro proc_lsst.multi INFO: found job 17153088 in provider astro 2024-03-17 07:06:39 proc_lsst.multi:201 [INFO] cancelling 17153088 on provider astro proc_lsst.multi INFO: cancelling 17153088 on provider astro 2024-03-17 07:06:39 proc_lsst.multi:146 [INFO] found job 17153125 in provider astro proc_lsst.multi INFO: found job 17153125 in provider astro 2024-03-17 07:06:39 proc_lsst.multi:201 [INFO] cancelling 17153125 on provider astro proc_lsst.multi INFO: cancelling 17153125 on provider astro 2024-03-17 07:06:40 proc_lsst.multi:146 [INFO] found job 17153120 in provider astro proc_lsst.multi INFO: found job 17153120 in provider astro 2024-03-17 07:06:40 proc_lsst.multi:201 [INFO] cancelling 17153120 on provider astro proc_lsst.multi INFO: cancelling 17153120 on provider astro 2024-03-17 07:06:40 proc_lsst.multi:146 [INFO] found job 17153123 in provider astro proc_lsst.multi INFO: found job 17153123 in provider astro 2024-03-17 07:06:40 proc_lsst.multi:201 [INFO] cancelling 17153123 on provider astro proc_lsst.multi INFO: cancelling 17153123 on provider astro 2024-03-17 07:06:40 proc_lsst.multi:146 [INFO] found job 17153121 in provider astro proc_lsst.multi INFO: found job 17153121 in provider astro 2024-03-17 07:06:40 proc_lsst.multi:201 [INFO] cancelling 17153121 on provider astro proc_lsst.multi INFO: cancelling 17153121 on provider astro 2024-03-17 07:06:40 proc_lsst.multi:146 [INFO] found job 17153098 in provider astro proc_lsst.multi INFO: found job 17153098 in provider astro 2024-03-17 07:06:40 proc_lsst.multi:201 [INFO] cancelling 17153098 on provider astro proc_lsst.multi INFO: cancelling 17153098 on provider astro 2024-03-17 07:06:41 proc_lsst.multi:146 [INFO] found job 17153151 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153151 in provider escience_rtx6k 2024-03-17 07:06:41 proc_lsst.multi:201 [INFO] cancelling 17153151 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153151 on provider escience_rtx6k parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-17 07:06:41 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-17 07:06:41 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17153088', '17153089', '17153090', '17153091', '17153093', '17153094', '17153096', '17153097', '17153098', '17153099', '17153100', '17153101', '17153103', '17153104', '17153106', '17153108', '17153110', '17153115', '17153117', '17153118', '17153119', '17153120', '17153121', '17153123', '17153125', '17153127', '17153129', '17153135', '17153137', '17153140', '17153153', '17153155', '17153157', '17153159', '17153161', '17153163', '17153165', '17153167', '17153169', '17153171', '17153173', '17153142', '17153145', '17153146', '17153149', '17153151', '53768'] proc_lsst.multi INFO: new jobs since last cancel ['17153088', '17153089', '17153090', '17153091', '17153093', '17153094', '17153096', '17153097', '17153098', '17153099', '17153100', '17153101', '17153103', '17153104', '17153106', '17153108', '17153110', '17153115', '17153117', '17153118', '17153119', '17153120', '17153121', '17153123', '17153125', '17153127', '17153129', '17153135', '17153137', '17153140', '17153153', '17153155', '17153157', '17153159', '17153161', '17153163', '17153165', '17153167', '17153169', '17153171', '17153173', '17153142', '17153145', '17153146', '17153149', '17153151', '53768'] 2024-03-17 07:06:41 proc_lsst.multi:146 [INFO] found job 17153088 in provider astro proc_lsst.multi INFO: found job 17153088 in provider astro 2024-03-17 07:06:41 proc_lsst.multi:201 [INFO] cancelling 17153088 on provider astro proc_lsst.multi INFO: cancelling 17153088 on provider astro 2024-03-17 07:06:41 proc_lsst.multi:146 [INFO] found job 17153089 in provider astro proc_lsst.multi INFO: found job 17153089 in provider astro 2024-03-17 07:06:41 proc_lsst.multi:201 [INFO] cancelling 17153089 on provider astro proc_lsst.multi INFO: cancelling 17153089 on provider astro 2024-03-17 07:06:41 proc_lsst.multi:146 [INFO] found job 17153090 in provider astro proc_lsst.multi INFO: found job 17153090 in provider astro 2024-03-17 07:06:41 proc_lsst.multi:201 [INFO] cancelling 17153090 on provider astro proc_lsst.multi INFO: cancelling 17153090 on provider astro 2024-03-17 07:06:42 proc_lsst.multi:146 [INFO] found job 17153091 in provider astro proc_lsst.multi INFO: found job 17153091 in provider astro 2024-03-17 07:06:42 proc_lsst.multi:201 [INFO] cancelling 17153091 on provider astro proc_lsst.multi INFO: cancelling 17153091 on provider astro 2024-03-17 07:06:42 proc_lsst.multi:146 [INFO] found job 17153093 in provider astro proc_lsst.multi INFO: found job 17153093 in provider astro 2024-03-17 07:06:42 proc_lsst.multi:201 [INFO] cancelling 17153093 on provider astro proc_lsst.multi INFO: cancelling 17153093 on provider astro 2024-03-17 07:06:42 proc_lsst.multi:146 [INFO] found job 17153094 in provider astro proc_lsst.multi INFO: found job 17153094 in provider astro 2024-03-17 07:06:42 proc_lsst.multi:201 [INFO] cancelling 17153094 on provider astro proc_lsst.multi INFO: cancelling 17153094 on provider astro 2024-03-17 07:06:42 proc_lsst.multi:146 [INFO] found job 17153096 in provider astro proc_lsst.multi INFO: found job 17153096 in provider astro 2024-03-17 07:06:42 proc_lsst.multi:201 [INFO] cancelling 17153096 on provider astro proc_lsst.multi INFO: cancelling 17153096 on provider astro 2024-03-17 07:06:43 proc_lsst.multi:146 [INFO] found job 17153097 in provider astro proc_lsst.multi INFO: found job 17153097 in provider astro 2024-03-17 07:06:43 proc_lsst.multi:201 [INFO] cancelling 17153097 on provider astro proc_lsst.multi INFO: cancelling 17153097 on provider astro 2024-03-17 07:06:43 proc_lsst.multi:146 [INFO] found job 17153098 in provider astro proc_lsst.multi INFO: found job 17153098 in provider astro 2024-03-17 07:06:43 proc_lsst.multi:201 [INFO] cancelling 17153098 on provider astro proc_lsst.multi INFO: cancelling 17153098 on provider astro 2024-03-17 07:06:44 proc_lsst.multi:146 [INFO] found job 17153099 in provider astro proc_lsst.multi INFO: found job 17153099 in provider astro 2024-03-17 07:06:44 proc_lsst.multi:201 [INFO] cancelling 17153099 on provider astro proc_lsst.multi INFO: cancelling 17153099 on provider astro 2024-03-17 07:06:44 proc_lsst.multi:146 [INFO] found job 17153100 in provider astro proc_lsst.multi INFO: found job 17153100 in provider astro 2024-03-17 07:06:44 proc_lsst.multi:201 [INFO] cancelling 17153100 on provider astro proc_lsst.multi INFO: cancelling 17153100 on provider astro 2024-03-17 07:06:44 proc_lsst.multi:146 [INFO] found job 17153101 in provider astro proc_lsst.multi INFO: found job 17153101 in provider astro 2024-03-17 07:06:44 proc_lsst.multi:201 [INFO] cancelling 17153101 on provider astro proc_lsst.multi INFO: cancelling 17153101 on provider astro 2024-03-17 07:06:44 proc_lsst.multi:146 [INFO] found job 17153103 in provider astro proc_lsst.multi INFO: found job 17153103 in provider astro 2024-03-17 07:06:44 proc_lsst.multi:201 [INFO] cancelling 17153103 on provider astro proc_lsst.multi INFO: cancelling 17153103 on provider astro 2024-03-17 07:06:45 proc_lsst.multi:146 [INFO] found job 17153104 in provider astro proc_lsst.multi INFO: found job 17153104 in provider astro 2024-03-17 07:06:45 proc_lsst.multi:201 [INFO] cancelling 17153104 on provider astro proc_lsst.multi INFO: cancelling 17153104 on provider astro 2024-03-17 07:06:45 proc_lsst.multi:146 [INFO] found job 17153106 in provider astro proc_lsst.multi INFO: found job 17153106 in provider astro 2024-03-17 07:06:45 proc_lsst.multi:201 [INFO] cancelling 17153106 on provider astro proc_lsst.multi INFO: cancelling 17153106 on provider astro 2024-03-17 07:06:45 proc_lsst.multi:146 [INFO] found job 17153108 in provider astro proc_lsst.multi INFO: found job 17153108 in provider astro 2024-03-17 07:06:45 proc_lsst.multi:201 [INFO] cancelling 17153108 on provider astro proc_lsst.multi INFO: cancelling 17153108 on provider astro 2024-03-17 07:06:46 proc_lsst.multi:146 [INFO] found job 17153110 in provider astro proc_lsst.multi INFO: found job 17153110 in provider astro 2024-03-17 07:06:46 proc_lsst.multi:201 [INFO] cancelling 17153110 on provider astro proc_lsst.multi INFO: cancelling 17153110 on provider astro 2024-03-17 07:06:46 proc_lsst.multi:146 [INFO] found job 17153115 in provider astro proc_lsst.multi INFO: found job 17153115 in provider astro 2024-03-17 07:06:46 proc_lsst.multi:201 [INFO] cancelling 17153115 on provider astro proc_lsst.multi INFO: cancelling 17153115 on provider astro 2024-03-17 07:06:46 proc_lsst.multi:146 [INFO] found job 17153117 in provider astro proc_lsst.multi INFO: found job 17153117 in provider astro 2024-03-17 07:06:46 proc_lsst.multi:201 [INFO] cancelling 17153117 on provider astro proc_lsst.multi INFO: cancelling 17153117 on provider astro 2024-03-17 07:06:46 proc_lsst.multi:146 [INFO] found job 17153118 in provider astro proc_lsst.multi INFO: found job 17153118 in provider astro 2024-03-17 07:06:46 proc_lsst.multi:201 [INFO] cancelling 17153118 on provider astro proc_lsst.multi INFO: cancelling 17153118 on provider astro 2024-03-17 07:06:47 proc_lsst.multi:146 [INFO] found job 17153119 in provider astro proc_lsst.multi INFO: found job 17153119 in provider astro 2024-03-17 07:06:47 proc_lsst.multi:201 [INFO] cancelling 17153119 on provider astro proc_lsst.multi INFO: cancelling 17153119 on provider astro 2024-03-17 07:06:47 proc_lsst.multi:146 [INFO] found job 17153120 in provider astro proc_lsst.multi INFO: found job 17153120 in provider astro 2024-03-17 07:06:47 proc_lsst.multi:201 [INFO] cancelling 17153120 on provider astro proc_lsst.multi INFO: cancelling 17153120 on provider astro 2024-03-17 07:06:47 proc_lsst.multi:146 [INFO] found job 17153121 in provider astro proc_lsst.multi INFO: found job 17153121 in provider astro 2024-03-17 07:06:47 proc_lsst.multi:201 [INFO] cancelling 17153121 on provider astro proc_lsst.multi INFO: cancelling 17153121 on provider astro 2024-03-17 07:06:47 proc_lsst.multi:146 [INFO] found job 17153123 in provider astro proc_lsst.multi INFO: found job 17153123 in provider astro 2024-03-17 07:06:47 proc_lsst.multi:201 [INFO] cancelling 17153123 on provider astro proc_lsst.multi INFO: cancelling 17153123 on provider astro 2024-03-17 07:06:48 proc_lsst.multi:146 [INFO] found job 17153125 in provider astro proc_lsst.multi INFO: found job 17153125 in provider astro 2024-03-17 07:06:48 proc_lsst.multi:201 [INFO] cancelling 17153125 on provider astro proc_lsst.multi INFO: cancelling 17153125 on provider astro 2024-03-17 07:06:48 proc_lsst.multi:146 [INFO] found job 17153127 in provider astro proc_lsst.multi INFO: found job 17153127 in provider astro 2024-03-17 07:06:48 proc_lsst.multi:201 [INFO] cancelling 17153127 on provider astro proc_lsst.multi INFO: cancelling 17153127 on provider astro 2024-03-17 07:06:48 proc_lsst.multi:146 [INFO] found job 17153129 in provider astro proc_lsst.multi INFO: found job 17153129 in provider astro 2024-03-17 07:06:48 proc_lsst.multi:201 [INFO] cancelling 17153129 on provider astro proc_lsst.multi INFO: cancelling 17153129 on provider astro 2024-03-17 07:06:48 proc_lsst.multi:146 [INFO] found job 17153135 in provider astro proc_lsst.multi INFO: found job 17153135 in provider astro 2024-03-17 07:06:48 proc_lsst.multi:201 [INFO] cancelling 17153135 on provider astro proc_lsst.multi INFO: cancelling 17153135 on provider astro 2024-03-17 07:06:49 proc_lsst.multi:146 [INFO] found job 17153137 in provider astro proc_lsst.multi INFO: found job 17153137 in provider astro 2024-03-17 07:06:49 proc_lsst.multi:201 [INFO] cancelling 17153137 on provider astro proc_lsst.multi INFO: cancelling 17153137 on provider astro 2024-03-17 07:06:49 proc_lsst.multi:146 [INFO] found job 17153140 in provider astro proc_lsst.multi INFO: found job 17153140 in provider astro 2024-03-17 07:06:49 proc_lsst.multi:201 [INFO] cancelling 17153140 on provider astro proc_lsst.multi INFO: cancelling 17153140 on provider astro 2024-03-17 07:06:49 proc_lsst.multi:146 [INFO] found job 17153153 in provider escience_a40 proc_lsst.multi INFO: found job 17153153 in provider escience_a40 2024-03-17 07:06:49 proc_lsst.multi:201 [INFO] cancelling 17153153 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153153 on provider escience_a40 2024-03-17 07:06:49 proc_lsst.multi:146 [INFO] found job 17153155 in provider escience_a40 proc_lsst.multi INFO: found job 17153155 in provider escience_a40 2024-03-17 07:06:49 proc_lsst.multi:201 [INFO] cancelling 17153155 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153155 on provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:146 [INFO] found job 17153157 in provider escience_a40 proc_lsst.multi INFO: found job 17153157 in provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:201 [INFO] cancelling 17153157 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153157 on provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:146 [INFO] found job 17153159 in provider escience_a40 proc_lsst.multi INFO: found job 17153159 in provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:201 [INFO] cancelling 17153159 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153159 on provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:146 [INFO] found job 17153161 in provider escience_a40 proc_lsst.multi INFO: found job 17153161 in provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:201 [INFO] cancelling 17153161 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153161 on provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:146 [INFO] found job 17153163 in provider escience_a40 proc_lsst.multi INFO: found job 17153163 in provider escience_a40 2024-03-17 07:06:50 proc_lsst.multi:201 [INFO] cancelling 17153163 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153163 on provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:146 [INFO] found job 17153165 in provider escience_a40 proc_lsst.multi INFO: found job 17153165 in provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:201 [INFO] cancelling 17153165 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153165 on provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:146 [INFO] found job 17153167 in provider escience_a40 proc_lsst.multi INFO: found job 17153167 in provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:201 [INFO] cancelling 17153167 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153167 on provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:146 [INFO] found job 17153169 in provider escience_a40 proc_lsst.multi INFO: found job 17153169 in provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:201 [INFO] cancelling 17153169 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153169 on provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:146 [INFO] found job 17153171 in provider escience_a40 proc_lsst.multi INFO: found job 17153171 in provider escience_a40 2024-03-17 07:06:51 proc_lsst.multi:201 [INFO] cancelling 17153171 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153171 on provider escience_a40 2024-03-17 07:06:52 proc_lsst.multi:146 [INFO] found job 17153173 in provider escience_a40 proc_lsst.multi INFO: found job 17153173 in provider escience_a40 2024-03-17 07:06:52 proc_lsst.multi:201 [INFO] cancelling 17153173 on provider escience_a40 proc_lsst.multi INFO: cancelling 17153173 on provider escience_a40 2024-03-17 07:06:52 proc_lsst.multi:146 [INFO] found job 17153142 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153142 in provider escience_rtx6k 2024-03-17 07:06:52 proc_lsst.multi:201 [INFO] cancelling 17153142 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153142 on provider escience_rtx6k 2024-03-17 07:06:52 proc_lsst.multi:146 [INFO] found job 17153145 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153145 in provider escience_rtx6k 2024-03-17 07:06:52 proc_lsst.multi:201 [INFO] cancelling 17153145 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153145 on provider escience_rtx6k 2024-03-17 07:06:52 proc_lsst.multi:146 [INFO] found job 17153146 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153146 in provider escience_rtx6k 2024-03-17 07:06:52 proc_lsst.multi:201 [INFO] cancelling 17153146 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153146 on provider escience_rtx6k 2024-03-17 07:06:53 proc_lsst.multi:146 [INFO] found job 17153149 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153149 in provider escience_rtx6k 2024-03-17 07:06:53 proc_lsst.multi:201 [INFO] cancelling 17153149 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153149 on provider escience_rtx6k 2024-03-17 07:06:53 proc_lsst.multi:146 [INFO] found job 17153151 in provider escience_rtx6k proc_lsst.multi INFO: found job 17153151 in provider escience_rtx6k 2024-03-17 07:06:53 proc_lsst.multi:201 [INFO] cancelling 17153151 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17153151 on provider escience_rtx6k 2024-03-17 07:06:53 proc_lsst.multi:146 [INFO] found job 53768 in provider local proc_lsst.multi INFO: found job 53768 in provider local 2024-03-17 07:06:53 proc_lsst.multi:201 [INFO] cancelling 53768 on provider local proc_lsst.multi INFO: cancelling 53768 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53768 parsl.providers.local.local WARNING: Failed to kill PID: 53768 and child processes on local 2024-03-17 07:06:54 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 136.9828 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210909_B1b_science#step0_20240317T070413Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 136.9916 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 156.6897 seconds; current memory usage: 0.272 Gibyte, delta: 0.135 Gibyte, peak delta: 0.137 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.274 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210909_B1b_science#step0_20240317T070413Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m38.789s user 0m32.631s sys 0m13.727s