+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210510/A1c --output-run DEEP/20210510/A1c/science#step0/20240315T100328Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.2470 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0043 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0095 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/pipeline_orig.qgraph DEEP/20210510/A1c/science#step0/20240315T100328Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.7098 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.2917 seconds; current memory usage: 0.272 Gibyte, delta: 0.094 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0147 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210510_A1c_science#step0_20240315T100328Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0831 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2297 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.265 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 3cd7c9c1-ba41-49b4-b0df-d148748be2da parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-15 10:03:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-15 10:03:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 10:03:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-15 10:03:51 proc_lsst.multi:170 [INFO] [multi] job_id 56858 proc_lsst.multi INFO: [multi] job_id 56858 2024-03-15 10:03:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:51 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 56858 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 56858 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 56858 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 882d852db76e1ff48783b9cceb3a66aa parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/35bf5d40-dba6-4368-a692-e4d4bbbdbcae_isrForCrosstalkSources_995992_39.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/35bf5d40-dba6-4368-a692-e4d4bbbdbcae_isrForCrosstalkSources_995992_39.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 340293b9e2ea72378774257ae2ee8d9b parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d92f2779-73d2-4fd5-966e-f5df6e105921_isrForCrosstalkSources_995992_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d92f2779-73d2-4fd5-966e-f5df6e105921_isrForCrosstalkSources_995992_42.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash ea65083dc4b609b7d4bc830c34719623 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5c935824-e56a-4821-adf0-69dc2543d6ec_isrForCrosstalkSources_995847_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5c935824-e56a-4821-adf0-69dc2543d6ec_isrForCrosstalkSources_995847_47.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 4d8cae11ced154b0214f3686a49af50c parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/835f257c-d8ac-4d8a-986f-4c231da08b9f_isrForCrosstalkSources_995991_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/835f257c-d8ac-4d8a-986f-4c231da08b9f_isrForCrosstalkSources_995991_15.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash a6371145eccaba370e7b54cc87e94961 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c4260cb5-4b20-41a8-a399-ea8cfed632a6_isrForCrosstalkSources_995991_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c4260cb5-4b20-41a8-a399-ea8cfed632a6_isrForCrosstalkSources_995991_42.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 5e8166402508d9c26e58cca563f594e4 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d558a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cd020327-ff4f-44a2-8df4-c026a8f5badd_isrForCrosstalkSources_995845_12.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cd020327-ff4f-44a2-8df4-c026a8f5badd_isrForCrosstalkSources_995845_12.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 1477d573205724312a456b1d84a70ff5 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3db351f1-39b1-4c44-8da8-675a47d1c5f8_isrForCrosstalkSources_995845_48.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3db351f1-39b1-4c44-8da8-675a47d1c5f8_isrForCrosstalkSources_995845_48.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 897ef4eb2c86007864215bb4b2e67b0a parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d54c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/463ff919-0ded-43a0-aba5-31c4cc570b19_isrForCrosstalkSources_995991_59.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/463ff919-0ded-43a0-aba5-31c4cc570b19_isrForCrosstalkSources_995991_59.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 3a521560dadbde1d453ad7f3abd89098 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d8aa8680-49a1-4cee-8e31-f3a12226a887_isrForCrosstalkSources_995991_56.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d8aa8680-49a1-4cee-8e31-f3a12226a887_isrForCrosstalkSources_995991_56.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash ee1afe3e85c2774b3f5e29df0938a10e parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d554e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d7dda995-0445-4070-88a8-ae7e353ad480_isrForCrosstalkSources_995846_15.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d7dda995-0445-4070-88a8-ae7e353ad480_isrForCrosstalkSources_995846_15.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 3e4d38a62b97c960a40ff8a296a44679 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d559e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fab65104-e10c-48ab-a38e-5bc4735cb2a1_isrForCrosstalkSources_995845_58.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fab65104-e10c-48ab-a38e-5bc4735cb2a1_isrForCrosstalkSources_995845_58.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash e4f7d16699142e9879a21e8eeff3a8b9 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ea214338-3e35-4ea6-8718-757415e4ead3_isrForCrosstalkSources_995846_25.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ea214338-3e35-4ea6-8718-757415e4ead3_isrForCrosstalkSources_995846_25.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash afa6fd0b77db0d7d51ba38d7d0d61e39 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/27285127-9683-470f-86d6-75f24b0fefbb_isrForCrosstalkSources_995991_60.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/27285127-9683-470f-86d6-75f24b0fefbb_isrForCrosstalkSources_995991_60.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash edb7b91c0cf7191e5ab096210b224688 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/11937b21-5244-4962-8328-8f7b5747fd3b_isrForCrosstalkSources_995991_35.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/11937b21-5244-4962-8328-8f7b5747fd3b_isrForCrosstalkSources_995991_35.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 0ded2119edcb770e6616d53ad0a65179 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/90dd4cc7-6723-4b93-bad3-6ff383d60b99_isrForCrosstalkSources_995991_31.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/90dd4cc7-6723-4b93-bad3-6ff383d60b99_isrForCrosstalkSources_995991_31.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 43f0a79e00d0432e2ebd513b8052ffde parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d55ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ff6460f8-1fa4-4313-85b9-02782fa24783_isrForCrosstalkSources_995846_48.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ff6460f8-1fa4-4313-85b9-02782fa24783_isrForCrosstalkSources_995846_48.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 93e28ef2bcd3864c502298f2020a3291 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d563e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2536cfa3-9266-4098-b02c-ba2d6f72fb93_isrForCrosstalkSources_995991_22.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2536cfa3-9266-4098-b02c-ba2d6f72fb93_isrForCrosstalkSources_995991_22.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 3c3840400dc29636a08a3debeac2b8a3 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d565c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/323ce1bd-eb67-46c6-bdfe-8f47c674a22a_isrForCrosstalkSources_995991_49.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/323ce1bd-eb67-46c6-bdfe-8f47c674a22a_isrForCrosstalkSources_995991_49.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash e3622ab1ed8a9de53f8a57a059a1963a parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/41c0209d-30e7-4bfc-8adb-0e9bb9dae078_isrForCrosstalkSources_995991_36.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/41c0209d-30e7-4bfc-8adb-0e9bb9dae078_isrForCrosstalkSources_995991_36.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 6d8b32b7524cf86641d90ba6f5c81209 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df051932-a21a-4fd8-bfc3-1415e70c417b_isrForCrosstalkSources_995846_1.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df051932-a21a-4fd8-bfc3-1415e70c417b_isrForCrosstalkSources_995846_1.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash d3889d21b5809c2424e63a1c71552ba3 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d562a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5c633328-a323-4fc9-80c2-efa611f9c215_isrForCrosstalkSources_995846_11.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5c633328-a323-4fc9-80c2-efa611f9c215_isrForCrosstalkSources_995846_11.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 406621d0f3cf7e6301c89eb617d065a6 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93b18cdc-9a59-4c90-b422-59c1a09ccbd0_isrForCrosstalkSources_995992_53.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93b18cdc-9a59-4c90-b422-59c1a09ccbd0_isrForCrosstalkSources_995992_53.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 536e7197dfc0c5870ace0f035484816a parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c991212b-1b80-4835-b814-3e8f2be3a6a0_isrForCrosstalkSources_995991_50.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c991212b-1b80-4835-b814-3e8f2be3a6a0_isrForCrosstalkSources_995991_50.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 54126c7b107ff90738048cc1ef7e10af parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d567a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3d5fb016-f33c-4298-a70a-0e7ea7ba03be_isrForCrosstalkSources_995847_1.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3d5fb016-f33c-4298-a70a-0e7ea7ba03be_isrForCrosstalkSources_995847_1.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 8de61b723ebb561f3512555f08b59a79 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c417b168-0e47-49e8-a0a7-5cac45927923_isrForCrosstalkSources_995990_34.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c417b168-0e47-49e8-a0a7-5cac45927923_isrForCrosstalkSources_995990_34.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash bb2ca3e09a2cf7f5644618523c89654b parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/35094275-4daa-4ff3-bb79-95e46a6320f4_isrForCrosstalkSources_995847_14.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/35094275-4daa-4ff3-bb79-95e46a6320f4_isrForCrosstalkSources_995847_14.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 212a3ad76bf17ec5d7e640628b5a8c99 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d568e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7cd437ab-7efa-48aa-8d84-a2bb1bbb8b33_isrForCrosstalkSources_995990_9.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7cd437ab-7efa-48aa-8d84-a2bb1bbb8b33_isrForCrosstalkSources_995990_9.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash c91c66c5e1a8e415cc25ca645732bb0f parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/20bec065-4fc3-4575-87de-e1bf16941cf4_isrForCrosstalkSources_995991_18.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/20bec065-4fc3-4575-87de-e1bf16941cf4_isrForCrosstalkSources_995991_18.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 3d70aea15ced0d5f600cb1d64f50e32c parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f08887bf-c684-4744-aa94-ca4c9e890e4b_isrForCrosstalkSources_995991_7.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f08887bf-c684-4744-aa94-ca4c9e890e4b_isrForCrosstalkSources_995991_7.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 23d652f82915634e014ee876204741dc parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/21817f50-7180-47e7-bb67-c4856ed9c96b_isrForCrosstalkSources_995846_42.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/21817f50-7180-47e7-bb67-c4856ed9c96b_isrForCrosstalkSources_995846_42.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash cb208db5c2c85bebf17d8c5867cd08be parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/76257e55-ddd6-47cd-b506-46ceada84227_isrForCrosstalkSources_995992_32.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/76257e55-ddd6-47cd-b506-46ceada84227_isrForCrosstalkSources_995992_32.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 2a697ebae7d74e0841aa987a9ea9d7ae parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/16914a28-d0e5-49e6-b38d-47af2a84618c_isrForCrosstalkSources_995845_56.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/16914a28-d0e5-49e6-b38d-47af2a84618c_isrForCrosstalkSources_995845_56.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 4a23c05c4f774c041a9746588c0add75 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/b0668813-4351-473f-a349-5bf6a941c6ab_isrForCrosstalkSources_995845_61.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/b0668813-4351-473f-a349-5bf6a941c6ab_isrForCrosstalkSources_995845_61.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 9190e16f00ae9b8e7d45d787770d4c8d parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/85812fdd-c377-4ea8-8585-8ba02f24a37b_isrForCrosstalkSources_995990_19.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/85812fdd-c377-4ea8-8585-8ba02f24a37b_isrForCrosstalkSources_995990_19.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash aa5e561f06dfd3fa369f389474c08c6f parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/cc8853a9-2983-4cb7-a46d-d5a0ebac42f8_isrForCrosstalkSources_995991_23.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/cc8853a9-2983-4cb7-a46d-d5a0ebac42f8_isrForCrosstalkSources_995991_23.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 21081e7cf8bd072355b39d47fe1db7a2 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fa427cf6-e1dd-4357-b65b-76e63376a5ed_isrForCrosstalkSources_995992_2.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fa427cf6-e1dd-4357-b65b-76e63376a5ed_isrForCrosstalkSources_995992_2.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 633e904678d2ca7281581b482a565bf8 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2f75b453-387a-4809-99c7-1246eed11093_isrForCrosstalkSources_995846_58.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2f75b453-387a-4809-99c7-1246eed11093_isrForCrosstalkSources_995846_58.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 2913f38f004be34d48c988074df8637b parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d571a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/661ae5b1-ebce-4719-b5e8-844d39a61806_isrForCrosstalkSources_995847_44.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/661ae5b1-ebce-4719-b5e8-844d39a61806_isrForCrosstalkSources_995847_44.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 36349037e7ae106e0042032c84b47282 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c6bbb435-04ae-4631-96e9-d0ab0b7a9f36_isrForCrosstalkSources_995847_46.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c6bbb435-04ae-4631-96e9-d0ab0b7a9f36_isrForCrosstalkSources_995847_46.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 2eb9dc42ad0cd95ac1ee7b198e8d2231 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a329cae9-d65a-4b91-aa97-da15bdc5b440_isrForCrosstalkSources_995990_3.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a329cae9-d65a-4b91-aa97-da15bdc5b440_isrForCrosstalkSources_995990_3.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 4f8620bf09692e45f8227bb8d55befaa parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/04bf41da-fb5f-4c46-96ff-16a1b8d516c4_isrForCrosstalkSources_995847_53.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/04bf41da-fb5f-4c46-96ff-16a1b8d516c4_isrForCrosstalkSources_995847_53.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash f59e7f5ad2f49d207141b4b30d35c3d4 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e3a27af-3543-4606-b2cb-9fc14ed868bd_isrForCrosstalkSources_995992_34.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e3a27af-3543-4606-b2cb-9fc14ed868bd_isrForCrosstalkSources_995992_34.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 557b68d0f8aca8d261e0974d675d7a79 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/937c8b74-5dd1-494f-bc01-105f043f1e61_isrForCrosstalkSources_995846_39.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/937c8b74-5dd1-494f-bc01-105f043f1e61_isrForCrosstalkSources_995846_39.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash c24b09c1e33d89ba8ce1702126a5223e parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/da315e9d-90d7-4560-a47c-1a0adfef95ca_isrForCrosstalkSources_995846_14.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/da315e9d-90d7-4560-a47c-1a0adfef95ca_isrForCrosstalkSources_995846_14.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 9ca2bf230b0a48e9217f6611a7739eff parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/30777447-e809-4cfa-a076-5757ca018936_isrForCrosstalkSources_995990_31.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/30777447-e809-4cfa-a076-5757ca018936_isrForCrosstalkSources_995990_31.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 97105c9052ed99d779bdb62f43d715a3 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f5da9a35-2cbc-4558-bbe3-210434c7161a_isrForCrosstalkSources_995845_14.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f5da9a35-2cbc-4558-bbe3-210434c7161a_isrForCrosstalkSources_995845_14.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash f8ed6f3a9953bea297e97b95695877d2 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3b85802a-2f08-48cc-a065-2c518f3d62ab_isrForCrosstalkSources_995990_55.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3b85802a-2f08-48cc-a065-2c518f3d62ab_isrForCrosstalkSources_995990_55.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 1f8ca953d3c4a333e94303160c608d06 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d579c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d569e4a4-1f81-43c5-be03-8a05b32292bb_isrForCrosstalkSources_995992_54.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d569e4a4-1f81-43c5-be03-8a05b32292bb_isrForCrosstalkSources_995992_54.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 5db3cf2f48930c2da09b182e69a698d0 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7cbf917-3638-47d6-b0da-3af6294c8231_isrForCrosstalkSources_995992_33.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7cbf917-3638-47d6-b0da-3af6294c8231_isrForCrosstalkSources_995992_33.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 30a5c64745fce5afaed6b8b044b30efd parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d577e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b9540e2c-2ca0-4718-97fc-10eaedda582e_isrForCrosstalkSources_995991_55.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b9540e2c-2ca0-4718-97fc-10eaedda582e_isrForCrosstalkSources_995991_55.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash cb60e4f1c5522493a241fc35eaf631ed parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fe4624ea-529b-4161-8998-17917dbd35d6_isrForCrosstalkSources_995992_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fe4624ea-529b-4161-8998-17917dbd35d6_isrForCrosstalkSources_995992_27.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash a00253860774208a3ea39a6cc149c57a parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c202bfe6-b2bf-448a-856d-9fd244bf249b_isrForCrosstalkSources_995990_25.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c202bfe6-b2bf-448a-856d-9fd244bf249b_isrForCrosstalkSources_995990_25.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 153a61f1563891f798f68d9960183ed6 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d576a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/336e0a8d-045b-42d2-adc1-8b315077ca82_isrForCrosstalkSources_995845_8.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/336e0a8d-045b-42d2-adc1-8b315077ca82_isrForCrosstalkSources_995845_8.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash f949fa409d31a1973bac2c397fa21147 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ee7deebf-64d0-4c3a-99ca-85dbc8afd71d_isrForCrosstalkSources_995845_33.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ee7deebf-64d0-4c3a-99ca-85dbc8afd71d_isrForCrosstalkSources_995845_33.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 602189faf01df3561181394e4c1b7467 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d574c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9400445c-d5b1-48b8-8d2c-c39df1ea3347_isrForCrosstalkSources_995845_36.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9400445c-d5b1-48b8-8d2c-c39df1ea3347_isrForCrosstalkSources_995845_36.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash d266a1dfa7c8174ca0aeab706706da50 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5e2ac03d-4116-4826-85b0-e11db26ed7e6_isrForCrosstalkSources_995847_56.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5e2ac03d-4116-4826-85b0-e11db26ed7e6_isrForCrosstalkSources_995847_56.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 86de5156a2a261490ca9a7b56f5226fd parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c6fb5633-4d11-4713-965c-4affc39f2066_isrForCrosstalkSources_995992_40.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c6fb5633-4d11-4713-965c-4affc39f2066_isrForCrosstalkSources_995992_40.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 1cea83891f1c1ba57a4bebf975d4a985 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b449a9ad-e782-4f3c-8e39-712e09b8672b_isrForCrosstalkSources_995992_17.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b449a9ad-e782-4f3c-8e39-712e09b8672b_isrForCrosstalkSources_995992_17.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 67dd268e0dfc38b026280e65f650043c parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d572e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/6079bfd7-5b56-4d62-a726-4b1bab683e14_isrForCrosstalkSources_995990_39.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/6079bfd7-5b56-4d62-a726-4b1bab683e14_isrForCrosstalkSources_995990_39.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 5f74d4896a8b174414e6d7b829ede85a parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cf90d5f0-9d3e-4f3c-97d9-90c636b57f2c_isrForCrosstalkSources_995845_24.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cf90d5f0-9d3e-4f3c-97d9-90c636b57f2c_isrForCrosstalkSources_995845_24.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 4134f114c697cb86697c5b6bc9f78f42 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d56520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/bc994bfb-bfc7-41f6-ad70-122d3c28b96b_isrForCrosstalkSources_995847_35.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/bc994bfb-bfc7-41f6-ad70-122d3c28b96b_isrForCrosstalkSources_995847_35.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 09ee0cbae18ff32b3fa49ecf700ef4cc parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7000e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/65ed42de-02a8-47ae-bba3-9e8207c83bc0_isrForCrosstalkSources_995846_28.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/65ed42de-02a8-47ae-bba3-9e8207c83bc0_isrForCrosstalkSources_995846_28.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 9c954e7c8f44f83039e79e2ab6235774 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1c5a936a-0609-4f29-9d0b-c0d0c882500b_isrForCrosstalkSources_995847_59.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1c5a936a-0609-4f29-9d0b-c0d0c882500b_isrForCrosstalkSources_995847_59.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 1c07c6422883f77eb52edd07c8982c2b parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/59b0ce3e-7aac-43af-9c8b-1fae824e0bdb_isrForCrosstalkSources_995845_11.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/59b0ce3e-7aac-43af-9c8b-1fae824e0bdb_isrForCrosstalkSources_995845_11.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash b415eca1dcd16f4b1f27fbca51d21f13 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/721f5177-a9ba-40d5-9ff4-d7f013dfd2a8_isrForCrosstalkSources_995992_8.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/721f5177-a9ba-40d5-9ff4-d7f013dfd2a8_isrForCrosstalkSources_995992_8.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 252539e6fc685efe1d724db52e3e33c4 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/336f8516-2848-43b8-aac4-24bfc6602ba1_isrForCrosstalkSources_995992_9.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/336f8516-2848-43b8-aac4-24bfc6602ba1_isrForCrosstalkSources_995992_9.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash c3a85cdee6cfd5cff2d6e8cdb48c3037 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c3aead3e-748a-458e-a282-564ea3da94d6_isrForCrosstalkSources_995991_53.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c3aead3e-748a-458e-a282-564ea3da94d6_isrForCrosstalkSources_995991_53.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 0976a6d4f0c4ec80e0142c6b40aac6a7 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7004a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/eb24f8cc-d520-47cf-b228-bd85ef0f8ac8_isrForCrosstalkSources_995990_17.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/eb24f8cc-d520-47cf-b228-bd85ef0f8ac8_isrForCrosstalkSources_995990_17.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash f8cc566483658e952f5c34d84b7c35f6 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f06aea87-3b5f-40e8-ba80-606981ff3200_isrForCrosstalkSources_995991_5.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f06aea87-3b5f-40e8-ba80-606981ff3200_isrForCrosstalkSources_995991_5.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 5cf2e55c40fd5d521422395e27941782 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91d1b138-8375-41e2-985f-a55f3b0d4a49_isrForCrosstalkSources_995992_49.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91d1b138-8375-41e2-985f-a55f3b0d4a49_isrForCrosstalkSources_995992_49.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 545b5d79b80a71b67297efd94fde0e9e parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e66a9c1-e28f-435e-a01e-c2c7b885d67f_isrForCrosstalkSources_995992_20.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e66a9c1-e28f-435e-a01e-c2c7b885d67f_isrForCrosstalkSources_995992_20.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 4dbe24088e495ee57c1e34d4a975081b parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/8f101ac4-bc65-40b5-967f-4a12b541e3b5_isrForCrosstalkSources_995845_52.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/8f101ac4-bc65-40b5-967f-4a12b541e3b5_isrForCrosstalkSources_995845_52.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 4e8e3fe5cab63b11a6bd9cda2c6705ce parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7007c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/5d213017-a99d-4531-857f-d265a4f807e6_isrForCrosstalkSources_995992_1.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/5d213017-a99d-4531-857f-d265a4f807e6_isrForCrosstalkSources_995992_1.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash eb1083a83f1cd825b3d252923d1d2e07 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c92bb445-f3d0-44d6-8e2a-f52c63604aa5_isrForCrosstalkSources_995845_22.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c92bb445-f3d0-44d6-8e2a-f52c63604aa5_isrForCrosstalkSources_995845_22.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 97aebd40472502adca7af48f8dffd297 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e1b63eb7-7646-4222-b44f-13a05dc86041_isrForCrosstalkSources_995992_6.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e1b63eb7-7646-4222-b44f-13a05dc86041_isrForCrosstalkSources_995992_6.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash d9e304b3a996010b19f23ca10ceee88d parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7009a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/629e6089-9d0a-479f-8819-486bfefb148b_isrForCrosstalkSources_995991_32.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/629e6089-9d0a-479f-8819-486bfefb148b_isrForCrosstalkSources_995991_32.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 31141b045fbd19bbeac6d828a91fd12e parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/92c67fa7-2ca4-45fb-acc8-c734c20541a0_isrForCrosstalkSources_995846_23.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/92c67fa7-2ca4-45fb-acc8-c734c20541a0_isrForCrosstalkSources_995846_23.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash a46824d84d9b15cb58926e5ff9f1112f parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2dc2233b-1f38-44ad-8e50-fe5628de7432_isrForCrosstalkSources_995990_14.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2dc2233b-1f38-44ad-8e50-fe5628de7432_isrForCrosstalkSources_995990_14.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 6ab670539e453880f6691b4333502cd5 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1af821d6-06a2-424f-a4ee-bd076ddeff34_isrForCrosstalkSources_995845_27.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1af821d6-06a2-424f-a4ee-bd076ddeff34_isrForCrosstalkSources_995845_27.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 7026260eb545041d5adc9a263b1df329 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/20b3e70d-c965-49a0-83d8-c14ea52a3caf_isrForCrosstalkSources_995846_29.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/20b3e70d-c965-49a0-83d8-c14ea52a3caf_isrForCrosstalkSources_995846_29.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c0d81f9179a4b87e94379e744b9fb439 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5b526511-bfd2-404b-a536-ee1f646df749_isrForCrosstalkSources_995847_28.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5b526511-bfd2-404b-a536-ee1f646df749_isrForCrosstalkSources_995847_28.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash cdbd3d9afbcfa5f59001667fbdff760d parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07a47b58-89a7-48b5-ad83-08a6f751ec14_isrForCrosstalkSources_995845_60.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07a47b58-89a7-48b5-ad83-08a6f751ec14_isrForCrosstalkSources_995845_60.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 7a7b33ff59ea9824d395f7468f9fa1ce parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/55e10068-0527-44eb-adef-2224ecbb5253_isrForCrosstalkSources_995845_6.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/55e10068-0527-44eb-adef-2224ecbb5253_isrForCrosstalkSources_995845_6.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 6db6fada6aa628b56a1799ddf5a68257 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1f9cae6d-8f40-4d66-af8e-ecce2d254e98_isrForCrosstalkSources_995990_30.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1f9cae6d-8f40-4d66-af8e-ecce2d254e98_isrForCrosstalkSources_995990_30.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 5276cbba9e7d82ae09886c8bb1fa053e parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8748009d-99f2-4a2e-8a4a-73bf8f4ea840_isrForCrosstalkSources_995847_49.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8748009d-99f2-4a2e-8a4a-73bf8f4ea840_isrForCrosstalkSources_995847_49.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash bd2196bbbc0aba09ee10cad59d93adc4 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/282dd52a-9b83-4b30-8a07-dd556e5fa04a_isrForCrosstalkSources_995847_27.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/282dd52a-9b83-4b30-8a07-dd556e5fa04a_isrForCrosstalkSources_995847_27.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 3d81c788a8e5c14b0746d74f69a014a7 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/ee28cb78-a6b8-48f4-a910-4cf15c1231fd_isrForCrosstalkSources_995991_3.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/ee28cb78-a6b8-48f4-a910-4cf15c1231fd_isrForCrosstalkSources_995991_3.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 2f1d2bb534bc41cd7369ae59b9fec0b6 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bee75ebe-37e3-4593-b856-57da5d7e6d13_isrForCrosstalkSources_995990_22.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bee75ebe-37e3-4593-b856-57da5d7e6d13_isrForCrosstalkSources_995990_22.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 843381bc86bd7b591cc9cf4973bc8a24 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6a7d9a5c-eb6c-413e-ad5a-694b6c7d17d2_isrForCrosstalkSources_995846_13.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6a7d9a5c-eb6c-413e-ad5a-694b6c7d17d2_isrForCrosstalkSources_995846_13.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 0e2395565ccf1583a8d1040c69ef11de parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7011c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/565d0b86-fcca-4f4f-800a-c4cda4e2b766_isrForCrosstalkSources_995991_24.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/565d0b86-fcca-4f4f-800a-c4cda4e2b766_isrForCrosstalkSources_995991_24.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 60ccfc7be6eed18a7fd6a0f7b87a97c5 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2aa15bec-d236-4fc8-9c81-012d1e6083ad_isrForCrosstalkSources_995990_18.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2aa15bec-d236-4fc8-9c81-012d1e6083ad_isrForCrosstalkSources_995990_18.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash c90ab20fe658d3a82fe471e85e7549f1 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d9830d87-5ed6-4dc6-a511-cc7f8aa43b94_isrForCrosstalkSources_995991_28.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d9830d87-5ed6-4dc6-a511-cc7f8aa43b94_isrForCrosstalkSources_995991_28.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 5111c340e478469116ee3610287624b7 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/e2464fcd-30a1-458b-aaf2-253298a4c56d_isrForCrosstalkSources_995990_33.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/e2464fcd-30a1-458b-aaf2-253298a4c56d_isrForCrosstalkSources_995990_33.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 779b9fb3106903a22ee38a3bb12cd76c parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2e326dde-9d44-4994-8775-6fb0253a8554_isrForCrosstalkSources_995846_4.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2e326dde-9d44-4994-8775-6fb0253a8554_isrForCrosstalkSources_995846_4.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash ca94f3575a128ed8a4e9e2f94566ad24 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7014e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b3457f0b-4ecc-48f1-81ba-b6f3086ee24e_isrForCrosstalkSources_995847_21.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b3457f0b-4ecc-48f1-81ba-b6f3086ee24e_isrForCrosstalkSources_995847_21.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash ec6a24b233869b2c2527fbcbd3c28026 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f13141f4-1f67-4ac3-81d8-062e637033af_isrForCrosstalkSources_995845_54.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f13141f4-1f67-4ac3-81d8-062e637033af_isrForCrosstalkSources_995845_54.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 61de25258ed28aecd748681bebd66284 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7016c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e357c3ef-472a-49ae-8d5a-bab668983c61_isrForCrosstalkSources_995991_10.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e357c3ef-472a-49ae-8d5a-bab668983c61_isrForCrosstalkSources_995991_10.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash bc98bf161c8eba858abfced12abd9b79 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ff7bdd17-9f3e-4012-874f-6752d7c6393b_isrForCrosstalkSources_995845_16.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ff7bdd17-9f3e-4012-874f-6752d7c6393b_isrForCrosstalkSources_995845_16.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 57ff4d2650d42638e78eee9da646efdc parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/47399bfd-3ab3-4169-ab0e-dc975a7ce6b2_isrForCrosstalkSources_995845_44.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/47399bfd-3ab3-4169-ab0e-dc975a7ce6b2_isrForCrosstalkSources_995845_44.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 1683dd1a42958d9d4ada67f80c36d58d parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/da0526a1-4849-42c1-9f88-ac53aac773d2_isrForCrosstalkSources_995991_6.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/da0526a1-4849-42c1-9f88-ac53aac773d2_isrForCrosstalkSources_995991_6.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash e249959d243f68d875f056256004caab parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/307189b1-d742-4dde-a92a-5431248ff31b_isrForCrosstalkSources_995845_25.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/307189b1-d742-4dde-a92a-5431248ff31b_isrForCrosstalkSources_995845_25.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 132aac769c3b21ea505a8199ff93ea2f parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/39022dac-10ee-4f1e-ae53-5ea2f9f40119_isrForCrosstalkSources_995990_24.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/39022dac-10ee-4f1e-ae53-5ea2f9f40119_isrForCrosstalkSources_995990_24.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash fd6fa9ca83e4040e36a8a2e9a0fe8585 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/81f3f8ec-9981-4e89-831e-e26a6b97b5b3_isrForCrosstalkSources_995846_52.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/81f3f8ec-9981-4e89-831e-e26a6b97b5b3_isrForCrosstalkSources_995846_52.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash ce20e256dfa4c958c42cd95af5a1f075 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/846a4b2b-bd0b-4b91-ad5b-8d52a1cb3575_isrForCrosstalkSources_995990_35.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/846a4b2b-bd0b-4b91-ad5b-8d52a1cb3575_isrForCrosstalkSources_995990_35.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash b7e882cf5af3db00feee87cfcaf02b4d parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/71237ccb-1659-40fc-872d-c020ffbd4a2d_isrForCrosstalkSources_995846_41.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/71237ccb-1659-40fc-872d-c020ffbd4a2d_isrForCrosstalkSources_995846_41.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 857c65918f1d4a1582864ef6c4cf6bb7 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/f07a590e-90a6-41d2-80ce-1013ff488b63_isrForCrosstalkSources_995846_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/f07a590e-90a6-41d2-80ce-1013ff488b63_isrForCrosstalkSources_995846_2.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 40eceb0a972459b2ea5a88bc9092dd15 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/c628900d-4f3b-4669-9b08-ee4fc466e6b5_isrForCrosstalkSources_995846_36.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/c628900d-4f3b-4669-9b08-ee4fc466e6b5_isrForCrosstalkSources_995846_36.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 46bffe20274bdf7505d697a6301ba296 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d3c0706-9594-430a-93bc-6f6c36b261ce_isrForCrosstalkSources_995846_62.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d3c0706-9594-430a-93bc-6f6c36b261ce_isrForCrosstalkSources_995846_62.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 0f3338bb50d409cac28c37c834682c28 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a620af50-aea3-4184-b623-f58d79bc0dec_isrForCrosstalkSources_995847_15.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a620af50-aea3-4184-b623-f58d79bc0dec_isrForCrosstalkSources_995847_15.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash ff234997fa665351bbc7757f545945c4 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/46a636f5-4c6f-4348-a1a4-3492310107ea_isrForCrosstalkSources_995992_23.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/46a636f5-4c6f-4348-a1a4-3492310107ea_isrForCrosstalkSources_995992_23.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 8ac4a2e43c024b6f5f47d01f616ce0e7 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b701f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5543756f-4617-4fd5-8e50-04428b550382_isrForCrosstalkSources_995845_35.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5543756f-4617-4fd5-8e50-04428b550382_isrForCrosstalkSources_995845_35.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 7b7d5477b28d4d67f193df3a79f5f74f parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3ed1ae41-dcd3-4987-951c-4205fc6105ff_isrForCrosstalkSources_995846_7.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3ed1ae41-dcd3-4987-951c-4205fc6105ff_isrForCrosstalkSources_995846_7.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 466f76d9e8e8577fef5b3751d29f2bb8 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7020c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2b7294da-3a8b-400c-be9f-6454789a21df_isrForCrosstalkSources_995992_26.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2b7294da-3a8b-400c-be9f-6454789a21df_isrForCrosstalkSources_995992_26.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 046a6111a412067c2cb2b87ee9dd23bd parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/92bc9d7a-d2af-4158-883e-abffd7008768_isrForCrosstalkSources_995845_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/92bc9d7a-d2af-4158-883e-abffd7008768_isrForCrosstalkSources_995845_2.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 9dfed459d3a2c2dce788781175bfd461 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3c7b42c1-e5ad-4f63-b92a-a5eec5339727_isrForCrosstalkSources_995845_43.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3c7b42c1-e5ad-4f63-b92a-a5eec5339727_isrForCrosstalkSources_995845_43.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash d06af8e2908d9fea32c528fdfea1bd20 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7022a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c624229a-ba25-480d-8bd2-316f425f09b7_isrForCrosstalkSources_995992_29.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c624229a-ba25-480d-8bd2-316f425f09b7_isrForCrosstalkSources_995992_29.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 8ef0e3dc7730f100c8ee1ee231f16388 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8e373c81-1296-4624-941d-2b13e5010119_isrForCrosstalkSources_995847_17.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8e373c81-1296-4624-941d-2b13e5010119_isrForCrosstalkSources_995847_17.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash ef1371d50c17f61e53391e791485abf4 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7023e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1ab3376-936a-4f47-b419-dd0cf19b2ffe_isrForCrosstalkSources_995846_5.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1ab3376-936a-4f47-b419-dd0cf19b2ffe_isrForCrosstalkSources_995846_5.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 28f658afb2e3551a4a6078707e1f238b parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/50d3c9eb-40c4-45a0-a3ef-1fe07378555a_isrForCrosstalkSources_995846_44.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/50d3c9eb-40c4-45a0-a3ef-1fe07378555a_isrForCrosstalkSources_995846_44.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 07ec2bf1c2ff934ccb8684470637cf72 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9858fe82-bf39-4f46-879f-ef7401e6481a_isrForCrosstalkSources_995845_18.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9858fe82-bf39-4f46-879f-ef7401e6481a_isrForCrosstalkSources_995845_18.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 4c4e8e57b361829bbdc7c3885cf36740 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7025c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/e1e5175d-db11-40f9-baea-93456d07e273_isrForCrosstalkSources_995845_9.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/e1e5175d-db11-40f9-baea-93456d07e273_isrForCrosstalkSources_995845_9.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 4b03ec063489064b1247808954290a92 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1c76213d-8c37-4256-8cdf-bfc81786afc2_isrForCrosstalkSources_995990_59.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1c76213d-8c37-4256-8cdf-bfc81786afc2_isrForCrosstalkSources_995990_59.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 92db638c828fa17a471fba33de31070b parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/84e2f9dd-a326-418d-a169-e3dc4c87adf8_isrForCrosstalkSources_995992_16.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/84e2f9dd-a326-418d-a169-e3dc4c87adf8_isrForCrosstalkSources_995992_16.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 2b4043987bd4eafb1297aee55003b07e parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c6be25ee-108b-4c61-92f6-2f79d9b4a749_isrForCrosstalkSources_995845_39.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c6be25ee-108b-4c61-92f6-2f79d9b4a749_isrForCrosstalkSources_995845_39.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 65b5fa39e4f2ae33403a6d9d95cc1be8 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7027a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9a741536-27b7-4eb3-903d-e40d72c35cc7_isrForCrosstalkSources_995845_29.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9a741536-27b7-4eb3-903d-e40d72c35cc7_isrForCrosstalkSources_995845_29.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash a6144bb5dfbeb3bd9ff080e68c42d2a7 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7028e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/790bfe18-1b9c-47fb-a2e6-10436247ea90_isrForCrosstalkSources_995990_8.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/790bfe18-1b9c-47fb-a2e6-10436247ea90_isrForCrosstalkSources_995990_8.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 8f2d3522d107ebec72eb0d76761b28b6 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/af9b46c2-5e0f-4a68-8fbc-80436ee9d728_isrForCrosstalkSources_995991_44.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/af9b46c2-5e0f-4a68-8fbc-80436ee9d728_isrForCrosstalkSources_995991_44.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 5cc0c6db63465c4ebf60cdc465f9492f parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/fb7b142d-6abc-4c30-9915-f6940ebc30d7_isrForCrosstalkSources_995847_38.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/fb7b142d-6abc-4c30-9915-f6940ebc30d7_isrForCrosstalkSources_995847_38.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash b5d715fadaa24b656dff2901658e2585 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/15086afe-c0dd-4249-a4b2-e031ddf0d04d_isrForCrosstalkSources_995990_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/15086afe-c0dd-4249-a4b2-e031ddf0d04d_isrForCrosstalkSources_995990_13.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 7d36aff7b66eb1f2c788680b32037afc parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/797d3164-7c6b-4c8a-8f6c-84dd6a37e62f_isrForCrosstalkSources_995847_33.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/797d3164-7c6b-4c8a-8f6c-84dd6a37e62f_isrForCrosstalkSources_995847_33.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 9283f57d83fe1f306954b1816e2c9826 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7e458e5f-3481-4276-8733-7b9fa78fdecf_isrForCrosstalkSources_995990_57.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7e458e5f-3481-4276-8733-7b9fa78fdecf_isrForCrosstalkSources_995990_57.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash e7a9db950c39277053b64f76b3a88e53 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a1435403-3a2e-4c80-a5ea-2016185eb67c_isrForCrosstalkSources_995990_43.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a1435403-3a2e-4c80-a5ea-2016185eb67c_isrForCrosstalkSources_995990_43.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash e4bc73515e175abc98ed2d98155b4818 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/992a093d-7da8-43be-a705-d80f38335c9f_isrForCrosstalkSources_995990_1.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/992a093d-7da8-43be-a705-d80f38335c9f_isrForCrosstalkSources_995990_1.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash f6cb60cd05237a8a6dd6b3402e24158b parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/278c6923-99f6-4d1a-be99-e1e2315439e3_isrForCrosstalkSources_995845_17.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/278c6923-99f6-4d1a-be99-e1e2315439e3_isrForCrosstalkSources_995845_17.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash e33e16e5e5383ebdd34ef883576a4a76 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fbfd1665-d593-4cef-9f68-ab8b9b9e8308_isrForCrosstalkSources_995990_7.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fbfd1665-d593-4cef-9f68-ab8b9b9e8308_isrForCrosstalkSources_995990_7.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash e0e1c31a6563b18253ebef4e300e5e16 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8d066233-e5fa-460e-8e30-14ae4387722e_isrForCrosstalkSources_995847_20.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8d066233-e5fa-460e-8e30-14ae4387722e_isrForCrosstalkSources_995847_20.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 14988f8df722c2ac08a52f496cdad72a parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b702fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/387760f1-cf1e-4959-aa57-a1df7df4fff8_isrForCrosstalkSources_995990_10.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/387760f1-cf1e-4959-aa57-a1df7df4fff8_isrForCrosstalkSources_995990_10.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 38db01c48bfbd34e6811c5317a647aac parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b00818bd-80a4-4cd0-97c6-aa54eb3abc25_isrForCrosstalkSources_995991_20.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b00818bd-80a4-4cd0-97c6-aa54eb3abc25_isrForCrosstalkSources_995991_20.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash b68bf11cec0607e7432dbc0b3eaefeab parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/82215ea4-f02a-4ef1-873a-baf98dbd0b2f_isrForCrosstalkSources_995845_45.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/82215ea4-f02a-4ef1-873a-baf98dbd0b2f_isrForCrosstalkSources_995845_45.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 09edbd5ada553e35b230c356fd6c8981 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/72c6f4e0-323d-4dbc-a415-549de9f2c4df_isrForCrosstalkSources_995990_38.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/72c6f4e0-323d-4dbc-a415-549de9f2c4df_isrForCrosstalkSources_995990_38.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash a1214b9c47ce17924e0084a6f6e30ced parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b0ac35a2-30f8-4230-a35e-fdae59dbd931_isrForCrosstalkSources_995992_10.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b0ac35a2-30f8-4230-a35e-fdae59dbd931_isrForCrosstalkSources_995992_10.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 6617bdb7f1965c6f59b94efdb25da6bf parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/81242608-706c-40e1-a38e-3cf08bf6f3ba_isrForCrosstalkSources_995991_27.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/81242608-706c-40e1-a38e-3cf08bf6f3ba_isrForCrosstalkSources_995991_27.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash b35f971a6bc8cc654466798b946e499d parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/876b9323-8193-4490-b6b6-f0a5416c8647_isrForCrosstalkSources_995846_9.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/876b9323-8193-4490-b6b6-f0a5416c8647_isrForCrosstalkSources_995846_9.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 68e9f4721b7165278c47ffedb97a79db parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 10:03:52 proc_lsst.multi:146 [INFO] found job 56858 in provider local proc_lsst.multi INFO: found job 56858 in provider local parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5e2e4382-774b-426c-818a-de63e2c18794_isrForCrosstalkSources_995845_3.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5e2e4382-774b-426c-818a-de63e2c18794_isrForCrosstalkSources_995845_3.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 26b71d7ac394c25cfe4f4e109439baf9 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e540d57240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/344b02b0-d64c-4649-a77e-670fbcf968bc_isrForCrosstalkSources_995846_22.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/344b02b0-d64c-4649-a77e-670fbcf968bc_isrForCrosstalkSources_995846_22.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash fbc66f7e3aff42daa182d2828f179bb9 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/d64bda82-f05d-4e3f-9de5-898ceab3d0e6_isrForCrosstalkSources_995847_54.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/d64bda82-f05d-4e3f-9de5-898ceab3d0e6_isrForCrosstalkSources_995847_54.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 8a963e2835d89428607622d83ccf4987 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c5839727-714f-4608-b478-1407897242a7_isrForCrosstalkSources_995990_32.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c5839727-714f-4608-b478-1407897242a7_isrForCrosstalkSources_995990_32.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 11067d649ef858883a3bc631fe97b3a4 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87d305fc-b60f-4f8e-b4c6-b50a18a39e8f_isrForCrosstalkSources_995847_4.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87d305fc-b60f-4f8e-b4c6-b50a18a39e8f_isrForCrosstalkSources_995847_4.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 3582b94b50d37e46f3b939dd9af14092 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d9d9fc3c-d301-4e24-bfff-e88853c471d2_isrForCrosstalkSources_995846_26.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d9d9fc3c-d301-4e24-bfff-e88853c471d2_isrForCrosstalkSources_995846_26.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash c84ffbe5be4f2397d94c309d43ff42cb parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a6e8572c-f375-4d06-a205-0c0077bc12d9_isrForCrosstalkSources_995846_49.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a6e8572c-f375-4d06-a205-0c0077bc12d9_isrForCrosstalkSources_995846_49.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 379c3415b11625fea610657ffbc2f38e parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbc48ee2-f4b0-467e-a5be-15697bfad714_isrForCrosstalkSources_995990_11.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbc48ee2-f4b0-467e-a5be-15697bfad714_isrForCrosstalkSources_995990_11.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 9b174fc225e56a6dfe00e5e347530610 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/f757448a-ac46-40e6-82b1-af738471ebdc_isrForCrosstalkSources_995990_29.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/f757448a-ac46-40e6-82b1-af738471ebdc_isrForCrosstalkSources_995990_29.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash c395eabdd83c6c4e625a9b5b37826265 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 150 parsl.dataflow.strategy DEBUG: Executor multi has 150 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-15 10:03:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-15 10:03:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 10:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/20037789-80e5-4f48-8c27-d976ab1c9905_isrForCrosstalkSources_995845_32.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/20037789-80e5-4f48-8c27-d976ab1c9905_isrForCrosstalkSources_995845_32.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 8f7569fbd5d06bd920603ca5b5d56d31 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b700040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b9ccc912-36c7-4f79-9227-dd75b487c75e_isrForCrosstalkSources_995992_57.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b9ccc912-36c7-4f79-9227-dd75b487c75e_isrForCrosstalkSources_995992_57.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 4c8a9dbc1e5c5e1c7ddf75ed78cf959a parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ec9239ef-1374-416d-ae17-afba24b47fb8_isrForCrosstalkSources_995847_29.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ec9239ef-1374-416d-ae17-afba24b47fb8_isrForCrosstalkSources_995847_29.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash ebb557a157d01268b49b8666f84a9238 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/edeb2679-825f-491c-8432-d3b5dc72aa41_isrForCrosstalkSources_995846_50.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/edeb2679-825f-491c-8432-d3b5dc72aa41_isrForCrosstalkSources_995846_50.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash fae1d4b977ed0d1059c911c599906337 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/999b96ff-d1d4-4ca9-a610-0c0931b2e60d_isrForCrosstalkSources_995847_43.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/999b96ff-d1d4-4ca9-a610-0c0931b2e60d_isrForCrosstalkSources_995847_43.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash d61d76ba775f1f08990fcfbe99936ace parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7632e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1b12c092-b9b7-453f-b8f5-764eba359194_isrForCrosstalkSources_995990_36.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1b12c092-b9b7-453f-b8f5-764eba359194_isrForCrosstalkSources_995990_36.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash dd07ff6f5322937a8a92ac76ed30f36b parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47473b2d-5205-4c2c-a7c5-f1e262726222_isrForCrosstalkSources_995990_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47473b2d-5205-4c2c-a7c5-f1e262726222_isrForCrosstalkSources_995990_61.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash e1ac83acfe3342b80308e300a9731dd0 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5aaed2f5-1a0e-415b-843b-677e8da988b1_isrForCrosstalkSources_995845_10.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5aaed2f5-1a0e-415b-843b-677e8da988b1_isrForCrosstalkSources_995845_10.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash fcff8977b1be037450921396859f4938 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cfbc892a-59a9-4c34-9ee5-d054c73561d3_isrForCrosstalkSources_995845_4.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cfbc892a-59a9-4c34-9ee5-d054c73561d3_isrForCrosstalkSources_995845_4.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 7b191335bb64b03f21e982096da3f665 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7600e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f9a38967-23be-41da-88bb-a785d6dfd49a_isrForCrosstalkSources_995992_13.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f9a38967-23be-41da-88bb-a785d6dfd49a_isrForCrosstalkSources_995992_13.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 93c33005a1e58d07184e1d4a4083326a parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7602c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d0151207-a303-41c9-81e3-1a634eec0d2c_isrForCrosstalkSources_995845_31.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d0151207-a303-41c9-81e3-1a634eec0d2c_isrForCrosstalkSources_995845_31.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 9481b62585091adf78069f00f6e119ea parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/48984ccc-1232-444c-8454-519f17dbf144_isrForCrosstalkSources_995846_59.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/48984ccc-1232-444c-8454-519f17dbf144_isrForCrosstalkSources_995846_59.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 7514ea35187563a5f82ca68728d399ee parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b1ea726-d581-4a43-9b7e-e1595b25e998_isrForCrosstalkSources_995847_10.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b1ea726-d581-4a43-9b7e-e1595b25e998_isrForCrosstalkSources_995847_10.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash c192288382b438ab20d87a2cd15f2b32 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d0458f3d-bd46-472f-aea1-5e6d917a5a23_isrForCrosstalkSources_995846_16.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d0458f3d-bd46-472f-aea1-5e6d917a5a23_isrForCrosstalkSources_995846_16.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 4279ebad791bead61e348997f61a49dc parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7604a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/550b5d96-6307-4c9b-99bb-6f83ec135ef5_isrForCrosstalkSources_995846_24.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/550b5d96-6307-4c9b-99bb-6f83ec135ef5_isrForCrosstalkSources_995846_24.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 5b06ddf32936f74fad926b0d792bcf2a parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ba38b8e7-3da0-4e3e-857d-9fe5954339c9_isrForCrosstalkSources_995990_40.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ba38b8e7-3da0-4e3e-857d-9fe5954339c9_isrForCrosstalkSources_995990_40.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 36b0a5165bb1af4b3141bd15980633c0 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d41e0ca-7e7b-402a-ac87-d56340029283_isrForCrosstalkSources_995846_40.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d41e0ca-7e7b-402a-ac87-d56340029283_isrForCrosstalkSources_995846_40.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 4e05a7e89634bd7e2e99df9fa8fd0bed parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7605e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4763d071-a06a-4bac-9fd5-4fdabeb1867b_isrForCrosstalkSources_995847_3.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4763d071-a06a-4bac-9fd5-4fdabeb1867b_isrForCrosstalkSources_995847_3.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 316ed3fdf37fd133ee166c6c6a8e2a02 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7607c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bc4002b9-08ea-437d-a611-3c8ad16ad7db_isrForCrosstalkSources_995990_20.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bc4002b9-08ea-437d-a611-3c8ad16ad7db_isrForCrosstalkSources_995990_20.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 5d25433553041aa377933945d70f6da2 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d0bace7-cc17-42ee-808a-007ad421fc74_isrForCrosstalkSources_995992_22.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d0bace7-cc17-42ee-808a-007ad421fc74_isrForCrosstalkSources_995992_22.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 8acf98311596e8140f4f75a710e2deb2 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/95319794-1484-4b48-ae8d-e7d5c5f14991_isrForCrosstalkSources_995990_51.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/95319794-1484-4b48-ae8d-e7d5c5f14991_isrForCrosstalkSources_995990_51.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 144e19a921ff9ee9a375b8fa617faee5 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7609a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1a70713b-fe95-4314-b038-842640d6a351_isrForCrosstalkSources_995991_39.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1a70713b-fe95-4314-b038-842640d6a351_isrForCrosstalkSources_995991_39.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 492b53c63e3dccefc1f84bb3c82ca67b parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fd7d70a3-5a6e-4c0e-b2e0-4f6caced8b55_isrForCrosstalkSources_995846_60.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fd7d70a3-5a6e-4c0e-b2e0-4f6caced8b55_isrForCrosstalkSources_995846_60.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 9051f25bf04e16fb280b83d989dc4c0a parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/68418a59-00dc-4958-9172-bc7f1ddd213b_isrForCrosstalkSources_995991_34.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/68418a59-00dc-4958-9172-bc7f1ddd213b_isrForCrosstalkSources_995991_34.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 9db151ceab8b5c7e27b20b9423552bc2 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/08e7b07f-5959-4a66-8d95-e366154928e7_isrForCrosstalkSources_995991_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/08e7b07f-5959-4a66-8d95-e366154928e7_isrForCrosstalkSources_995991_2.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 71e4e023b347e7529a278a824974f06b parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/939a2524-a544-43a2-a599-94c2f3df3ee4_isrForCrosstalkSources_995847_55.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/939a2524-a544-43a2-a599-94c2f3df3ee4_isrForCrosstalkSources_995847_55.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 3720c35a6e220c3efd5ca5dd79cc658c parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b22b242b-fa90-42d9-b703-b714c5dd70ff_isrForCrosstalkSources_995847_32.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b22b242b-fa90-42d9-b703-b714c5dd70ff_isrForCrosstalkSources_995847_32.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash a0c8a8bd478aa91af35cdab7b8dabfe7 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 10:03:52 proc_lsst.multi:170 [INFO] [multi] job_id 17063563 proc_lsst.multi INFO: [multi] job_id 17063563 2024-03-15 10:03:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063563 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063563 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17063563 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-15 10:03:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-15 10:03:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bfd5d3c7-c191-4d23-b30d-6a339417d2d4_isrForCrosstalkSources_995992_4.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bfd5d3c7-c191-4d23-b30d-6a339417d2d4_isrForCrosstalkSources_995992_4.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash c1719b24402804a80695ba1f02bfa457 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/553a4c9b-b22d-40ad-9ef6-34ecb1c39ea3_isrForCrosstalkSources_995847_25.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/553a4c9b-b22d-40ad-9ef6-34ecb1c39ea3_isrForCrosstalkSources_995847_25.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash c3a46a9a80f309b9bb4711526f00b998 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b8027155-d4e2-44d7-9fc8-7364cd73853c_isrForCrosstalkSources_995992_28.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b8027155-d4e2-44d7-9fc8-7364cd73853c_isrForCrosstalkSources_995992_28.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 9b89d6446ebbeafdc4032ba897093d4d parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/362509f6-f9a6-4548-b114-37a4b89ea9dd_isrForCrosstalkSources_995990_50.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/362509f6-f9a6-4548-b114-37a4b89ea9dd_isrForCrosstalkSources_995990_50.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash a4abc77536c5726810c30af9ac1c1e44 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/abd4fb7b-db27-494b-bd25-341e563ff9ad_isrForCrosstalkSources_995846_38.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/abd4fb7b-db27-494b-bd25-341e563ff9ad_isrForCrosstalkSources_995846_38.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 634300fd7a82f287dcbc419a6b8a6bb8 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/cafa834c-21a1-4a32-8d25-72ff43a35a76_isrForCrosstalkSources_995846_3.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/cafa834c-21a1-4a32-8d25-72ff43a35a76_isrForCrosstalkSources_995846_3.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash df511bc5fa059cff0e0949f1575761a5 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8787c5a6-761f-4830-9e77-b7d39e2f5299_isrForCrosstalkSources_995847_42.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8787c5a6-761f-4830-9e77-b7d39e2f5299_isrForCrosstalkSources_995847_42.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 4756cbc1c2e692f80386f23f1969a491 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c627918d-94ef-4b2a-9e7a-cd396cf442ba_isrForCrosstalkSources_995847_39.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c627918d-94ef-4b2a-9e7a-cd396cf442ba_isrForCrosstalkSources_995847_39.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 23e27d7f5547674cf142c34c45bbca54 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7611c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/70312bb8-3c0f-4299-a6b9-9e39ee9aa76d_isrForCrosstalkSources_995990_4.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/70312bb8-3c0f-4299-a6b9-9e39ee9aa76d_isrForCrosstalkSources_995990_4.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 3407176945fa075d1cf065d3ae933d4d parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/20de8146-7df2-476f-8319-6673d9ee78d4_isrForCrosstalkSources_995990_6.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/20de8146-7df2-476f-8319-6673d9ee78d4_isrForCrosstalkSources_995990_6.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 6855637603ef2b5bd88b7680b41d1670 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7613a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/74bf055e-09de-49e8-a6a7-38fbcc2cccf7_isrForCrosstalkSources_995990_37.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/74bf055e-09de-49e8-a6a7-38fbcc2cccf7_isrForCrosstalkSources_995990_37.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 271d7ded37e7a0cecf0306d709e989b9 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4a543e29-da7b-4a19-ac20-c0f704b2337b_isrForCrosstalkSources_995992_19.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4a543e29-da7b-4a19-ac20-c0f704b2337b_isrForCrosstalkSources_995992_19.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 8495a51f8fbfd9fb49f0b27e27fe7ef5 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d4ff9c6c-bc13-4c15-9023-7dc223322b83_isrForCrosstalkSources_995845_53.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d4ff9c6c-bc13-4c15-9023-7dc223322b83_isrForCrosstalkSources_995845_53.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash ecac84f2fb042b2537b91486b352fce8 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7ef89a59-7cee-4e30-8f6a-f644990202c9_isrForCrosstalkSources_995991_41.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7ef89a59-7cee-4e30-8f6a-f644990202c9_isrForCrosstalkSources_995991_41.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash a7dcd16b710bef49f572779aa9ccdb3a parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/9978b5c9-1776-400b-aa22-11ad065173f0_isrForCrosstalkSources_995846_43.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/9978b5c9-1776-400b-aa22-11ad065173f0_isrForCrosstalkSources_995846_43.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 857459503f4734f70ce579211415d9d9 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7616c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87a0c340-f872-41c0-a40d-977a90ddf653_isrForCrosstalkSources_995847_45.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87a0c340-f872-41c0-a40d-977a90ddf653_isrForCrosstalkSources_995847_45.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 3b5f6d8371824ec9c3998be6662a3b02 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7614e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3689b240-8380-4035-b900-7b88eea872bf_isrForCrosstalkSources_995847_41.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3689b240-8380-4035-b900-7b88eea872bf_isrForCrosstalkSources_995847_41.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash eeed284f0804c9cb3d592a50b7ebd7a8 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/14ebae4d-5611-4300-89cc-8ab243ecec76_isrForCrosstalkSources_995847_51.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/14ebae4d-5611-4300-89cc-8ab243ecec76_isrForCrosstalkSources_995847_51.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash c336afc84b6d9a183cf7333e47171b6f parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7618a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/4d29492c-e90e-4036-a3d1-ed0e39c402b8_isrForCrosstalkSources_995846_30.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/4d29492c-e90e-4036-a3d1-ed0e39c402b8_isrForCrosstalkSources_995846_30.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 3be5d40bdf4ea621edcd0ef6c810262d parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ee129b28-1af9-4596-a968-b6d7aca60791_isrForCrosstalkSources_995992_37.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ee129b28-1af9-4596-a968-b6d7aca60791_isrForCrosstalkSources_995992_37.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash c42504efc028c746f2a43485263dce8c parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/bacca6fa-3258-4d38-a58d-5c5c78eafa92_isrForCrosstalkSources_995991_30.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/bacca6fa-3258-4d38-a58d-5c5c78eafa92_isrForCrosstalkSources_995991_30.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash e8466560a8adbeedd6f13be11334f375 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fe8f70e3-651f-4050-97db-0394d30381c9_isrForCrosstalkSources_995845_7.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fe8f70e3-651f-4050-97db-0394d30381c9_isrForCrosstalkSources_995845_7.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 5a3b77235ef4764a4b03aaa6fbf12fac parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/a96a6d28-c90f-4694-b2c0-0837e5ce27c0_isrForCrosstalkSources_995845_49.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/a96a6d28-c90f-4694-b2c0-0837e5ce27c0_isrForCrosstalkSources_995845_49.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash d8b9ee27349fc45069f79e44f98ac4cf parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7619e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/311803fc-85ec-46a6-9ae8-2ed78a4b8752_isrForCrosstalkSources_995846_6.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/311803fc-85ec-46a6-9ae8-2ed78a4b8752_isrForCrosstalkSources_995846_6.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 8b8fc9226cf392d28ea8bef4537b116a parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/46e591fb-badd-4bea-89cd-53591914ad30_isrForCrosstalkSources_995991_25.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/46e591fb-badd-4bea-89cd-53591914ad30_isrForCrosstalkSources_995991_25.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash f125625e12713d48bd149226b65207b0 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ed77f56a-c121-482e-8ba1-5e6b57e96af0_isrForCrosstalkSources_995847_24.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ed77f56a-c121-482e-8ba1-5e6b57e96af0_isrForCrosstalkSources_995847_24.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash e73a21d19fe50dcd0a7899862978e2cd parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6897091e-3c7e-47ee-afb5-385decc32d17_isrForCrosstalkSources_995846_55.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6897091e-3c7e-47ee-afb5-385decc32d17_isrForCrosstalkSources_995846_55.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 185536ae5faa365c2d19bab7c9c42694 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 10:03:53 proc_lsst.multi:170 [INFO] [multi] job_id 17063564 proc_lsst.multi INFO: [multi] job_id 17063564 2024-03-15 10:03:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 10:03:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063564 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063564 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17063564 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-15 10:03:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-15 10:03:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 10:03:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/cae37bb9-9ae4-4f79-893c-6177f21008af_isrForCrosstalkSources_995990_56.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/cae37bb9-9ae4-4f79-893c-6177f21008af_isrForCrosstalkSources_995990_56.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 93aa24ae4dbeceba84d002512a68d613 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/36eb4233-14c2-488f-bd9e-42ff8d6c4826_isrForCrosstalkSources_995847_8.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/36eb4233-14c2-488f-bd9e-42ff8d6c4826_isrForCrosstalkSources_995847_8.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash a21fc23c6cbda3f70960323549b965f9 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/a91fd1e0-eee5-4868-8771-b00a8420e945_isrForCrosstalkSources_995991_57.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/a91fd1e0-eee5-4868-8771-b00a8420e945_isrForCrosstalkSources_995991_57.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 5650797d7c0908560444fe20c8b22c41 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/36b61023-ccc2-4326-ba0b-b49ca6e9bb70_isrForCrosstalkSources_995991_58.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/36b61023-ccc2-4326-ba0b-b49ca6e9bb70_isrForCrosstalkSources_995991_58.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash f2c1d0a5cab76a34621e9339b2b3bce9 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/37f5add3-0f05-4b8f-b825-ab77caa09031_isrForCrosstalkSources_995991_19.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/37f5add3-0f05-4b8f-b825-ab77caa09031_isrForCrosstalkSources_995991_19.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash cbb927c93fc9b0499a3fd5e3c0a280eb parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/83976f11-4141-46ea-bbaf-5c55b867954f_isrForCrosstalkSources_995992_48.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/83976f11-4141-46ea-bbaf-5c55b867954f_isrForCrosstalkSources_995992_48.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 44e4fa0c7ecaa0815f672501d8f9f88d parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3bf16483-dc11-41ec-ae9a-7fa224b4bd5a_isrForCrosstalkSources_995846_31.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3bf16483-dc11-41ec-ae9a-7fa224b4bd5a_isrForCrosstalkSources_995846_31.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash b86d35cb240ca31cc577d1b149aa7164 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47f20c36-a1ac-4878-9465-331cc452f900_isrForCrosstalkSources_995990_62.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47f20c36-a1ac-4878-9465-331cc452f900_isrForCrosstalkSources_995990_62.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash c4ccf488f1c7f069ff0d184e41b5d03a parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/12404c28-d096-4553-8c37-8ed600c7423b_isrForCrosstalkSources_995846_57.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/12404c28-d096-4553-8c37-8ed600c7423b_isrForCrosstalkSources_995846_57.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash c83d8824c7586691277abe4ed474ba0c parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/5acd44af-f6c1-48f0-bcbf-2f9a83e01544_isrForCrosstalkSources_995991_16.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/5acd44af-f6c1-48f0-bcbf-2f9a83e01544_isrForCrosstalkSources_995991_16.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash be382ee211aa921ed1a592c42ef5d75a parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/0f783341-ea76-496d-b22f-e6afd0a41970_isrForCrosstalkSources_995846_32.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/0f783341-ea76-496d-b22f-e6afd0a41970_isrForCrosstalkSources_995846_32.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash bdfc64fad7849566d0c08dc27164a350 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91430d59-bd32-46f1-a647-ce87f29c1533_isrForCrosstalkSources_995992_24.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91430d59-bd32-46f1-a647-ce87f29c1533_isrForCrosstalkSources_995992_24.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 0c29ecca8c62924ffc57f2b8280af543 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3276e957-6a6f-4587-b2aa-d0eab4c2e89e_isrForCrosstalkSources_995846_46.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3276e957-6a6f-4587-b2aa-d0eab4c2e89e_isrForCrosstalkSources_995846_46.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 1fd99dff29db1a0e111a772ca457aba5 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e5bc199d-de3b-49a9-9a19-65a8cef80f62_isrForCrosstalkSources_995991_21.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e5bc199d-de3b-49a9-9a19-65a8cef80f62_isrForCrosstalkSources_995991_21.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash efa352066924e56122684ff7a476d0d0 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/d0c73355-56ed-4adc-adcf-3c1a2eea55d3_isrForCrosstalkSources_995990_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/d0c73355-56ed-4adc-adcf-3c1a2eea55d3_isrForCrosstalkSources_995990_2.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash bd014ddedf3391f00f4a395626485386 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/67fac9ff-3009-4bdf-a794-b5b540fc5ddd_isrForCrosstalkSources_995845_15.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/67fac9ff-3009-4bdf-a794-b5b540fc5ddd_isrForCrosstalkSources_995845_15.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash c2f6662be393e3f18eb65e20244bc2a1 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07c728af-3105-4215-9c4e-fc2ccf934446_isrForCrosstalkSources_995845_20.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07c728af-3105-4215-9c4e-fc2ccf934446_isrForCrosstalkSources_995845_20.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 1bcdd8f7a35a37f5ee56bba088ee94c2 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a3cd0ff-fe01-4843-a016-787312b112cf_isrForCrosstalkSources_995990_58.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a3cd0ff-fe01-4843-a016-787312b112cf_isrForCrosstalkSources_995990_58.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 3bffc608e54dbccb02cfff2cbace9845 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/4580211e-16b9-40a5-8f49-afaeb6ad7fbf_isrForCrosstalkSources_995845_55.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/4580211e-16b9-40a5-8f49-afaeb6ad7fbf_isrForCrosstalkSources_995845_55.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 531e53d9521d066b5ef914c1de621905 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a83470d-cdc6-4181-807a-a18b1c5ef8eb_isrForCrosstalkSources_995990_46.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a83470d-cdc6-4181-807a-a18b1c5ef8eb_isrForCrosstalkSources_995990_46.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash da2b6d82a9a21be85f198434b87fd0a5 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7628e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/49364ec9-0baa-4223-ae84-c1f7d9f7dbd5_isrForCrosstalkSources_995846_35.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/49364ec9-0baa-4223-ae84-c1f7d9f7dbd5_isrForCrosstalkSources_995846_35.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 123437e6a1467895630d75832026ff3b parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a0bca795-b524-442f-9070-386b79964360_isrForCrosstalkSources_995846_20.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a0bca795-b524-442f-9070-386b79964360_isrForCrosstalkSources_995846_20.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 4ceaf236f3309da3b283cc5c0faf26b1 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1c3561dc-664b-4a7e-ab65-52804d8e0110_isrForCrosstalkSources_995991_8.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1c3561dc-664b-4a7e-ab65-52804d8e0110_isrForCrosstalkSources_995991_8.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 00c1074b3228cff7fcbb19322a8a7b24 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7625c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/986a90b2-d4de-48a1-8df5-af854dc70fe2_isrForCrosstalkSources_995991_13.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/986a90b2-d4de-48a1-8df5-af854dc70fe2_isrForCrosstalkSources_995991_13.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash e3893542f851ae34e2edcb33ec2379b2 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/72137b87-d483-4dbd-bee3-9ae4421ccf0c_isrForCrosstalkSources_995992_18.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/72137b87-d483-4dbd-bee3-9ae4421ccf0c_isrForCrosstalkSources_995992_18.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 4d7af5a41b0561b9f57fd1c870b93f71 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7627a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/750c6ffd-8128-4f35-83ea-76f3245bf34f_isrForCrosstalkSources_995990_23.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/750c6ffd-8128-4f35-83ea-76f3245bf34f_isrForCrosstalkSources_995990_23.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 7f5ff24c3cf57fc771be8c629f4fea85 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/044c679d-49a1-49b0-90e7-b241de305794_isrForCrosstalkSources_995990_52.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/044c679d-49a1-49b0-90e7-b241de305794_isrForCrosstalkSources_995990_52.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash fb13326cee676e6535e3fb76c4daa737 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/53cb97ef-1194-4643-a713-4d9226e54ae9_isrForCrosstalkSources_995990_44.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/53cb97ef-1194-4643-a713-4d9226e54ae9_isrForCrosstalkSources_995990_44.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 84037d47f47dd36f82f48225dd30278f parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbbda821-5574-4470-8e72-9fed2e130169_isrForCrosstalkSources_995990_45.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbbda821-5574-4470-8e72-9fed2e130169_isrForCrosstalkSources_995990_45.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 468ce5dd55625e7f376bea12b012d1d6 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e9ecb920-fe13-44fa-ac6e-9b0f3a6e7d0e_isrForCrosstalkSources_995847_16.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e9ecb920-fe13-44fa-ac6e-9b0f3a6e7d0e_isrForCrosstalkSources_995847_16.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash ca36e8203ded787dc4aa725561139ac6 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/81afc037-3d06-41af-ab5b-33c7ed75d78a_isrForCrosstalkSources_995992_36.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/81afc037-3d06-41af-ab5b-33c7ed75d78a_isrForCrosstalkSources_995992_36.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 7325ffed6533baa40460ceb158d0e41a parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/4166b755-091d-47d8-88cc-13a36d110659_isrForCrosstalkSources_995990_26.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/4166b755-091d-47d8-88cc-13a36d110659_isrForCrosstalkSources_995990_26.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash f5c2728505b4e65cd80c4dca6b5be594 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/85101d00-c5b5-45ab-88c6-701f6266cfb2_isrForCrosstalkSources_995847_23.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/85101d00-c5b5-45ab-88c6-701f6266cfb2_isrForCrosstalkSources_995847_23.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash fd9018b48a3544de73233e1af9c23556 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/18e0eb21-5ffd-4284-bd62-6d2d517e5106_isrForCrosstalkSources_995991_17.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/18e0eb21-5ffd-4284-bd62-6d2d517e5106_isrForCrosstalkSources_995991_17.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash d9184e76974ab368683bfbf8f9942bc9 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/de58121d-da04-4607-9869-bd3119840cc2_isrForCrosstalkSources_995990_28.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/de58121d-da04-4607-9869-bd3119840cc2_isrForCrosstalkSources_995990_28.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash d4ab14ffe2c5912ec09adf0c2e61ecfa parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba59a3a1-eee0-41be-a50c-a1c062260f16_isrForCrosstalkSources_995847_2.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba59a3a1-eee0-41be-a50c-a1c062260f16_isrForCrosstalkSources_995847_2.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash b18f73de97bad4df295649ccc010475e parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d7e2a07a-fc13-42fd-8481-2ef080408d1f_isrForCrosstalkSources_995845_62.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d7e2a07a-fc13-42fd-8481-2ef080408d1f_isrForCrosstalkSources_995845_62.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash e1f26a8fcf4da136ea311837ceae38fb parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7639c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8b2d370-f5b3-4d03-b468-7892bccd9cef_isrForCrosstalkSources_995846_33.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8b2d370-f5b3-4d03-b468-7892bccd9cef_isrForCrosstalkSources_995846_33.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 7a8dd0641d9b0571e79f4515e565fe1b parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a23f5a-be22-4110-ae42-6fb728d1649e_isrForCrosstalkSources_995846_19.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a23f5a-be22-4110-ae42-6fb728d1649e_isrForCrosstalkSources_995846_19.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 3f519680f5501bbaeb304752db6cb9fa parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1f065fb2-09cb-4bab-9ae1-57f59399f7e4_isrForCrosstalkSources_995845_46.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1f065fb2-09cb-4bab-9ae1-57f59399f7e4_isrForCrosstalkSources_995845_46.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 990cdeed69e283cfa30730108f14c86f parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7637e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/50431d64-4e64-4414-9b19-0bb89dea3495_isrForCrosstalkSources_995992_58.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/50431d64-4e64-4414-9b19-0bb89dea3495_isrForCrosstalkSources_995992_58.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 64e6041717c306925621b2f47480ee94 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fb446150-0602-410d-93bb-d9948e612ca6_isrForCrosstalkSources_995992_55.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fb446150-0602-410d-93bb-d9948e612ca6_isrForCrosstalkSources_995992_55.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash 65efa6a13c1f19045870482794ad88e9 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7636a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2c94c9aa-eb75-4318-908c-79758e1c2c84_isrForCrosstalkSources_995846_45.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2c94c9aa-eb75-4318-908c-79758e1c2c84_isrForCrosstalkSources_995846_45.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 61e7ea882db102eceec1b31f37c7144e parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 10:03:53 proc_lsst.multi:170 [INFO] [multi] job_id 17063565 proc_lsst.multi INFO: [multi] job_id 17063565 2024-03-15 10:03:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 10:03:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063565 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063565 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17063565 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-15 10:03:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-15 10:03:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 10:03:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/afdb96a1-d173-4b46-805a-9439e4efd2f8_isrForCrosstalkSources_995847_58.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/afdb96a1-d173-4b46-805a-9439e4efd2f8_isrForCrosstalkSources_995847_58.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash e039e7880652a4dd3b71c3c6a21a0813 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7634c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9f087ac1-bd99-4a5e-97dd-961df316b244_isrForCrosstalkSources_995991_40.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9f087ac1-bd99-4a5e-97dd-961df316b244_isrForCrosstalkSources_995991_40.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 09b2ac7b9d8e4441dd29a9ccd2063f6a parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/864d7158-a0b0-464d-828d-a7f0c0432ad4_isrForCrosstalkSources_995991_46.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/864d7158-a0b0-464d-828d-a7f0c0432ad4_isrForCrosstalkSources_995991_46.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash f7ceaff7d45d418fe0e234b284690773 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b703ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2b565921-2d8b-4435-b5f5-46d4c6bf7bac_isrForCrosstalkSources_995846_51.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2b565921-2d8b-4435-b5f5-46d4c6bf7bac_isrForCrosstalkSources_995846_51.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash a3ddd3cd40cc4e5827d33ecc9b8fc6b7 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/63d52ae6-c85d-4801-8b2c-ab3aa691754d_isrForCrosstalkSources_995991_4.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/63d52ae6-c85d-4801-8b2c-ab3aa691754d_isrForCrosstalkSources_995991_4.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 8b77b45915810cef515dc5209c7bca2c parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a3ac5869-9bff-4b7f-a1c6-ab673140814d_isrForCrosstalkSources_995992_3.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a3ac5869-9bff-4b7f-a1c6-ab673140814d_isrForCrosstalkSources_995992_3.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 83e4788076573eaff865569b54fc54da parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/21d8ef43-9574-4697-8626-c5097d3ed8bf_isrForCrosstalkSources_995845_19.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/21d8ef43-9574-4697-8626-c5097d3ed8bf_isrForCrosstalkSources_995845_19.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 4ddf2bdcd96582173aab06ef687c3ce4 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7631a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/203b2ef9-c229-44d3-b152-180ef1e466df_isrForCrosstalkSources_995990_16.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/203b2ef9-c229-44d3-b152-180ef1e466df_isrForCrosstalkSources_995990_16.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 5092662e6556586b4a358a945c0d6e5e parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b760f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d1453187-fd81-4223-9e36-4a4b319da432_isrForCrosstalkSources_995845_28.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d1453187-fd81-4223-9e36-4a4b319da432_isrForCrosstalkSources_995845_28.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 79fcb7cff87a1e39a959f0e8069b8cb5 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b762160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7216422-beda-49cc-9e94-714c58d246e8_isrForCrosstalkSources_995992_51.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7216422-beda-49cc-9e94-714c58d246e8_isrForCrosstalkSources_995992_51.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 0b1d854804b9b1fe6d6414408abcccc2 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b761f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4e785b7a-7d80-4964-ad46-3749d62d625e_isrForCrosstalkSources_995991_51.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4e785b7a-7d80-4964-ad46-3749d62d625e_isrForCrosstalkSources_995991_51.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 9b7a641f958f7ff5635aed34d5508f1d parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7623e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f7a9fcf7-cc00-4395-887c-7d7ad72740c3_isrForCrosstalkSources_995992_35.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f7a9fcf7-cc00-4395-887c-7d7ad72740c3_isrForCrosstalkSources_995992_35.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 5b83b3fbbfb398bb1cb43fd815a6e832 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/faddacf8-8c5a-466c-8e80-878594863a0e_isrForCrosstalkSources_995991_26.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/faddacf8-8c5a-466c-8e80-878594863a0e_isrForCrosstalkSources_995991_26.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash ca2a4dcb7a3cfa1f17fd2acccc4c5eb9 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1a7c36a-e331-40df-aab4-c2897829668a_isrForCrosstalkSources_995846_47.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1a7c36a-e331-40df-aab4-c2897829668a_isrForCrosstalkSources_995846_47.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash cc7de3e48ebe4a92e3e50f7575f030c9 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c39f8360-42c4-4e79-a355-d26e4d4181b9_isrForCrosstalkSources_995990_27.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c39f8360-42c4-4e79-a355-d26e4d4181b9_isrForCrosstalkSources_995990_27.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash cf40134883be62c901d7149c51a2e03e parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/58de4f48-74ef-4592-8f88-ea1097cc190e_isrForCrosstalkSources_995991_52.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/58de4f48-74ef-4592-8f88-ea1097cc190e_isrForCrosstalkSources_995991_52.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 6a0e4f1435611a4a9f3dc2a45872060f parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7ca12956-745e-42a2-be58-c176051dc887_isrForCrosstalkSources_995992_14.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7ca12956-745e-42a2-be58-c176051dc887_isrForCrosstalkSources_995992_14.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 217d80b5c71d806014ee642242aeb429 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7c0f423c-d210-42a4-a917-a80e35dca88c_isrForCrosstalkSources_995990_12.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7c0f423c-d210-42a4-a917-a80e35dca88c_isrForCrosstalkSources_995990_12.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 8b2031a874906b3a54f802fb37e4e531 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/817e6812-649b-40fb-830f-c452c52575d6_isrForCrosstalkSources_995992_15.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/817e6812-649b-40fb-830f-c452c52575d6_isrForCrosstalkSources_995992_15.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 3f69d7059603916c028b05cfec5743eb parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c25df085-3130-4e27-9d95-8f867182ec66_isrForCrosstalkSources_995990_53.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c25df085-3130-4e27-9d95-8f867182ec66_isrForCrosstalkSources_995990_53.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 2a6ea95469ceb104f261eac272d97f26 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/63019885-9cbb-4633-ba08-6946905d041e_isrForCrosstalkSources_995992_62.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/63019885-9cbb-4633-ba08-6946905d041e_isrForCrosstalkSources_995992_62.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash e2c7b7e08f179d4df953004900cf8515 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fc982387-6609-4642-8771-3c166b84e386_isrForCrosstalkSources_995990_5.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fc982387-6609-4642-8771-3c166b84e386_isrForCrosstalkSources_995990_5.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 3fab091e125d36282f93579c83c236fc parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cda38f17-f7ae-41e0-8513-0a886f845ec0_isrForCrosstalkSources_995845_47.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cda38f17-f7ae-41e0-8513-0a886f845ec0_isrForCrosstalkSources_995845_47.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash f571188760965fada687de5e8c90376a parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/96a140dd-2d74-4976-8d8a-dbf2304b5a20_isrForCrosstalkSources_995991_38.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/96a140dd-2d74-4976-8d8a-dbf2304b5a20_isrForCrosstalkSources_995991_38.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 6535a0a8f9dec8bca8858f437c7394de parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ca366878-2efc-49b8-95cd-65850d491008_isrForCrosstalkSources_995990_47.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ca366878-2efc-49b8-95cd-65850d491008_isrForCrosstalkSources_995990_47.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 446f7cf1479d7160595b9239a10a1b01 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fabc346b-d535-4915-ba5c-378ebf9cdd1d_isrForCrosstalkSources_995846_27.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fabc346b-d535-4915-ba5c-378ebf9cdd1d_isrForCrosstalkSources_995846_27.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash e0400805d3ee3e92dd83cd81a231104c parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93213656-7d16-48a7-8211-3c69f74948ad_isrForCrosstalkSources_995992_41.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93213656-7d16-48a7-8211-3c69f74948ad_isrForCrosstalkSources_995992_41.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 691d386791d34aa1fa32bd1bfc635a01 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d2a175c1-4fa1-4955-8859-bccfd02db32e_isrForCrosstalkSources_995846_10.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d2a175c1-4fa1-4955-8859-bccfd02db32e_isrForCrosstalkSources_995846_10.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash f32b1c6594d9e7a2890ee81b1fba1110 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/02ff3113-9767-42ba-8ab8-4b74214c403a_isrForCrosstalkSources_995845_30.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/02ff3113-9767-42ba-8ab8-4b74214c403a_isrForCrosstalkSources_995845_30.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 34163fb0b2b94fc671f146db3a708277 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b7abfa2-a664-42aa-a495-f2ad3cef60bd_isrForCrosstalkSources_995847_60.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b7abfa2-a664-42aa-a495-f2ad3cef60bd_isrForCrosstalkSources_995847_60.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash c8cea64d7ff0f6d97ed86f03be235143 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3b5a15c2-a859-4940-a9ac-0106ba506c09_isrForCrosstalkSources_995992_50.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3b5a15c2-a859-4940-a9ac-0106ba506c09_isrForCrosstalkSources_995992_50.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 9c808eb58a3b8a3a8400510986033e49 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ced5a832-59b9-48a1-b527-388dbc3228de_isrForCrosstalkSources_995847_48.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ced5a832-59b9-48a1-b527-388dbc3228de_isrForCrosstalkSources_995847_48.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash c3987b77ed959265259602adcbd4df5a parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d760c2d0-f25c-4f6f-89a3-518e14ab2272_isrForCrosstalkSources_995992_59.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d760c2d0-f25c-4f6f-89a3-518e14ab2272_isrForCrosstalkSources_995992_59.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash cd7f04e147a3003a8d3423b8191a8c2b parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/843bb7df-e126-449f-8802-22872cb44908_isrForCrosstalkSources_995847_5.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/843bb7df-e126-449f-8802-22872cb44908_isrForCrosstalkSources_995847_5.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 15fd483f4ce49ae9b139ed6d7b5a6164 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/77cef725-4ac9-412d-93f0-e6dae111a4a1_isrForCrosstalkSources_995991_62.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/77cef725-4ac9-412d-93f0-e6dae111a4a1_isrForCrosstalkSources_995991_62.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 9bf626bcfa16e461fbfa193dc4a8f243 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69e2303c-6c75-433b-87fc-07b9c6a19436_isrForCrosstalkSources_995846_53.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69e2303c-6c75-433b-87fc-07b9c6a19436_isrForCrosstalkSources_995846_53.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 453264d37d9cb02fbd949ad41428b7e0 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/df364a44-805d-4e6e-9a0a-cdd322f4ca86_isrForCrosstalkSources_995991_47.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/df364a44-805d-4e6e-9a0a-cdd322f4ca86_isrForCrosstalkSources_995991_47.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 7179661a99e4f14d7c9a410c4aa65d81 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a1605ad8-bc4a-42f7-bebc-de8e4fa17e4e_isrForCrosstalkSources_995847_61.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a1605ad8-bc4a-42f7-bebc-de8e4fa17e4e_isrForCrosstalkSources_995847_61.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 074bc01d9b995358a23a8ca5324e3de5 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/48bcff86-f8d7-4718-a372-adf308de5569_isrForCrosstalkSources_995990_60.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/48bcff86-f8d7-4718-a372-adf308de5569_isrForCrosstalkSources_995990_60.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash c70509dbf98b5a10fd64a8b53e966df9 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6cd3da14-50c0-4d1b-bbf6-54a8ae887d48_isrForCrosstalkSources_995992_46.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6cd3da14-50c0-4d1b-bbf6-54a8ae887d48_isrForCrosstalkSources_995992_46.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 420ca511061afb5277f082861ba4c512 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/0ffedcda-d487-454b-b258-3033dc338bec_isrForCrosstalkSources_995991_11.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/0ffedcda-d487-454b-b258-3033dc338bec_isrForCrosstalkSources_995991_11.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash cd50702db754937baa663b544b227240 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/0508290b-311f-41c3-818b-c81c923ee83e_isrForCrosstalkSources_995990_49.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/0508290b-311f-41c3-818b-c81c923ee83e_isrForCrosstalkSources_995990_49.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 71f388d12e62a3d52ad93ecbda7f9530 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7f8e638e-3d0e-48b4-9ef8-b6de66c5a09a_isrForCrosstalkSources_995991_45.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7f8e638e-3d0e-48b4-9ef8-b6de66c5a09a_isrForCrosstalkSources_995991_45.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 5d8fa28fc0f32dff9718ae181961f88e parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3a355815-87eb-4a4b-8c22-3191789ee3f9_isrForCrosstalkSources_995990_15.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3a355815-87eb-4a4b-8c22-3191789ee3f9_isrForCrosstalkSources_995990_15.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 2cc35c0829e02c296c7e6001033ae289 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/efe25367-f651-4904-8570-1d8a8f90d608_isrForCrosstalkSources_995845_37.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/efe25367-f651-4904-8570-1d8a8f90d608_isrForCrosstalkSources_995845_37.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 2eccf731dcfa536e8252769581077936 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9a283d20-f1ba-484c-8098-9bfd8e50b3e9_isrForCrosstalkSources_995847_30.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9a283d20-f1ba-484c-8098-9bfd8e50b3e9_isrForCrosstalkSources_995847_30.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash b8e247af83486f00c68037ac6e613f7e parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/9e3ef6f2-83b4-4d8e-bd52-4b8ed204b9d9_isrForCrosstalkSources_995992_60.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/9e3ef6f2-83b4-4d8e-bd52-4b8ed204b9d9_isrForCrosstalkSources_995992_60.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash c1cc69472972fbed38d6ac915ad4c9c6 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/9ed339d3-27d7-4961-a253-212c9664cb45_isrForCrosstalkSources_995990_41.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/9ed339d3-27d7-4961-a253-212c9664cb45_isrForCrosstalkSources_995990_41.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 648ac96c4761937cb34c8ba3c4f87d92 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/135e2f36-bd6d-4fc4-8350-991c5f95e855_isrForCrosstalkSources_995990_21.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/135e2f36-bd6d-4fc4-8350-991c5f95e855_isrForCrosstalkSources_995990_21.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash ba2e3ad942a6b921e1c375a689913599 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/806ff80f-5ba6-4bea-a319-a3f006da5f0d_isrForCrosstalkSources_995992_30.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/806ff80f-5ba6-4bea-a319-a3f006da5f0d_isrForCrosstalkSources_995992_30.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 0aa7f87b5e32218af4eb9d9e353f2526 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1e327ac7-7368-49f7-aac8-b8a6c5b636c1_isrForCrosstalkSources_995991_9.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1e327ac7-7368-49f7-aac8-b8a6c5b636c1_isrForCrosstalkSources_995991_9.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash f61308a277608a4e73704966ff50aebb parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/dc851891-5103-4228-92fe-b1e7b242750c_isrForCrosstalkSources_995845_42.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/dc851891-5103-4228-92fe-b1e7b242750c_isrForCrosstalkSources_995845_42.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 3b9f77b21cbfd6a6ebf01b889301b7ef parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3dedd72b-3b16-4880-85a1-463074353a77_isrForCrosstalkSources_995990_42.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3dedd72b-3b16-4880-85a1-463074353a77_isrForCrosstalkSources_995990_42.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 36c9047143fb649aef7ba1dc2d1061da parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5da31c73-6d89-43b7-9806-dea62dbf38ba_isrForCrosstalkSources_995846_18.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5da31c73-6d89-43b7-9806-dea62dbf38ba_isrForCrosstalkSources_995846_18.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 3d1460edb9efc40ec49bae164932b711 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b7abea17-7a89-47ef-bc7e-fca11d3b76e8_isrForCrosstalkSources_995847_12.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b7abea17-7a89-47ef-bc7e-fca11d3b76e8_isrForCrosstalkSources_995847_12.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash ee5cd960cf61ed3eea7086257adcd0ca parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2c1fe54c-4fdd-4743-8b41-b633a018946e_isrForCrosstalkSources_995991_1.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2c1fe54c-4fdd-4743-8b41-b633a018946e_isrForCrosstalkSources_995991_1.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 6a47de161085e859e16f392d31da7cee parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df6892ec-435f-447c-9df3-8cedcedb127b_isrForCrosstalkSources_995846_21.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df6892ec-435f-447c-9df3-8cedcedb127b_isrForCrosstalkSources_995846_21.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 5cc3da7255ff4cb49baf6735450e43a5 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a0cb0f-db5e-4636-aaff-03195b7e5cc3_isrForCrosstalkSources_995846_61.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a0cb0f-db5e-4636-aaff-03195b7e5cc3_isrForCrosstalkSources_995846_61.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 7a85bbadf700515763e18dee37151682 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/045d9106-edae-4e92-a3c3-81baeede3a08_isrForCrosstalkSources_995992_44.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/045d9106-edae-4e92-a3c3-81baeede3a08_isrForCrosstalkSources_995992_44.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 82e5fd33fed6d7410348a1499934a756 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7beca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/304b72ba-1249-4726-81a6-17e343932d65_isrForCrosstalkSources_995992_45.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/304b72ba-1249-4726-81a6-17e343932d65_isrForCrosstalkSources_995992_45.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash e39f0430099cde66324906c3dc25ac5e parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/92ebdacd-bc3b-4038-945f-08d9a53024bd_isrForCrosstalkSources_995847_31.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/92ebdacd-bc3b-4038-945f-08d9a53024bd_isrForCrosstalkSources_995847_31.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 150453ceca7be4c199f00a1afd6a14c1 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/97c81f40-079c-4f5b-a649-148b0f92f3a7_isrForCrosstalkSources_995847_13.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/97c81f40-079c-4f5b-a649-148b0f92f3a7_isrForCrosstalkSources_995847_13.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 1813f3afe470cbe0d8abb82bcea10621 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7beb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b733804a-8abf-438f-96b7-c575fc18e202_isrForCrosstalkSources_995847_50.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b733804a-8abf-438f-96b7-c575fc18e202_isrForCrosstalkSources_995847_50.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 6386b90b09a84d600c946a5d10fdd280 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7beac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/52496e44-5dbb-4f5e-b574-f1c9a8c3dba5_isrForCrosstalkSources_995845_23.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/52496e44-5dbb-4f5e-b574-f1c9a8c3dba5_isrForCrosstalkSources_995845_23.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash e27c807e0b301883f922927ca3ce635f parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/50a9214b-ea08-493a-a8ec-22dad3babcd1_isrForCrosstalkSources_995847_18.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/50a9214b-ea08-493a-a8ec-22dad3babcd1_isrForCrosstalkSources_995847_18.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 32f4d86f3be53e7fc5035ec33cbbba3f parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/7443d55d-4203-4283-a897-637513683281_isrForCrosstalkSources_995845_40.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/7443d55d-4203-4283-a897-637513683281_isrForCrosstalkSources_995845_40.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 5de46735d7f7cf7cec4dc692d783d680 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/eec3fab4-222a-4b5a-883f-b631ceb60bb8_isrForCrosstalkSources_995845_34.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/eec3fab4-222a-4b5a-883f-b631ceb60bb8_isrForCrosstalkSources_995845_34.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 2145ba78f90e7ab029a4d25abc179188 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c0da0851-b476-4ffc-b4ef-d1503e5d4eb3_isrForCrosstalkSources_995991_61.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c0da0851-b476-4ffc-b4ef-d1503e5d4eb3_isrForCrosstalkSources_995991_61.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 9d7e8e1f49fc5c3320095cb44f7e6163 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1d81c678-c569-47e3-8664-ed44706e37a5_isrForCrosstalkSources_995847_40.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1d81c678-c569-47e3-8664-ed44706e37a5_isrForCrosstalkSources_995847_40.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 6984f25d9bc49f1a809a1286b13c8536 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 10:03:54 proc_lsst.multi:170 [INFO] [multi] job_id 17063566 proc_lsst.multi INFO: [multi] job_id 17063566 2024-03-15 10:03:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 10:03:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063566 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063566 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17063566 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-15 10:03:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-15 10:03:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 10:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2e2300ea-ba8f-4f70-83b6-7a5f2b9f733b_isrForCrosstalkSources_995992_61.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2e2300ea-ba8f-4f70-83b6-7a5f2b9f733b_isrForCrosstalkSources_995992_61.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 77e409c315b76ada6777af5ff9135b07 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7620c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/814d0c6c-51e4-4daf-a4a3-62086fb25133_isrForCrosstalkSources_995845_21.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/814d0c6c-51e4-4daf-a4a3-62086fb25133_isrForCrosstalkSources_995845_21.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash cd1616d7cfe9bf84b1ba661992e814be parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7622a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f0fa09ac-9315-4daa-a43b-988264e8e151_isrForCrosstalkSources_995845_57.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f0fa09ac-9315-4daa-a43b-988264e8e151_isrForCrosstalkSources_995845_57.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 9c3bf82d222047a201b936e5d997f271 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b763600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9c992dc4-8552-4ab4-a05a-c394b0cef12c_isrForCrosstalkSources_995847_57.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9c992dc4-8552-4ab4-a05a-c394b0cef12c_isrForCrosstalkSources_995847_57.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 2e267e87be876aeab4e1fc79a6abd441 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bebcd5c3-5a50-4d5a-8b3d-5dc791e2fbf8_isrForCrosstalkSources_995992_7.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bebcd5c3-5a50-4d5a-8b3d-5dc791e2fbf8_isrForCrosstalkSources_995992_7.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 7ba7f35cc2a2d10ec7655ede32f0da39 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e0bcbb12-f44b-49d9-a508-131748fe47f8_isrForCrosstalkSources_995847_22.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e0bcbb12-f44b-49d9-a508-131748fe47f8_isrForCrosstalkSources_995847_22.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash ae4a7b140385575ea9bf724cbef0a365 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7befc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b45d01b1-cad1-4d3f-9089-5474479b24d9_isrForCrosstalkSources_995991_14.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b45d01b1-cad1-4d3f-9089-5474479b24d9_isrForCrosstalkSources_995991_14.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 31888a535c76e00690f3f1806d3325dd parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9e06a40f-85fc-4c1a-9c23-e003380e2608_isrForCrosstalkSources_995991_48.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9e06a40f-85fc-4c1a-9c23-e003380e2608_isrForCrosstalkSources_995991_48.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash c6b36607740155a90bf65980810fd56c parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/bb88c9bd-294a-4b6a-9260-cda938e99388_isrForCrosstalkSources_995846_54.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/bb88c9bd-294a-4b6a-9260-cda938e99388_isrForCrosstalkSources_995846_54.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 41c5a73104050df19b8aef44f0c7496b parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/26494850-a6fb-4665-ad39-8a957ce2fbdf_isrForCrosstalkSources_995845_41.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/26494850-a6fb-4665-ad39-8a957ce2fbdf_isrForCrosstalkSources_995845_41.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash b141a27e994aae1d4d8cfa41c430c47b parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7acc35a0-40c7-4ef3-9842-346c8789975d_isrForCrosstalkSources_995992_12.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7acc35a0-40c7-4ef3-9842-346c8789975d_isrForCrosstalkSources_995992_12.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash cc0521abafced793d7b948ad27d037fb parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4888c786-7f64-4ea7-8fc8-b19e91f491ca_isrForCrosstalkSources_995991_33.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4888c786-7f64-4ea7-8fc8-b19e91f491ca_isrForCrosstalkSources_995991_33.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 3b979f1ee59b0711c63c13ee41ea4109 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/382f9849-e4db-470b-af5b-ee5e3da176b8_isrForCrosstalkSources_995847_37.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/382f9849-e4db-470b-af5b-ee5e3da176b8_isrForCrosstalkSources_995847_37.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 51285ab62a55307e2589de65c0e8b860 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/90d68a8e-4665-4f62-a46d-fe6cce5b3bee_isrForCrosstalkSources_995990_54.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/90d68a8e-4665-4f62-a46d-fe6cce5b3bee_isrForCrosstalkSources_995990_54.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 64f3dabecb36e4b9bf8dd1f4368898b4 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3023bbf5-1470-48d3-9aa7-fcd43740c881_isrForCrosstalkSources_995847_52.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3023bbf5-1470-48d3-9aa7-fcd43740c881_isrForCrosstalkSources_995847_52.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash e08b5ccee2ba62393984af153302bdfc parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b6bfa1f5-9b39-4ea7-8bd4-9f2f4bceb9e1_isrForCrosstalkSources_995847_62.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b6bfa1f5-9b39-4ea7-8bd4-9f2f4bceb9e1_isrForCrosstalkSources_995847_62.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 11d48b0afa07d12b2ff110616fa3fdf2 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f4c94185-263e-4d4b-8772-19a7312f64b4_isrForCrosstalkSources_995991_12.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f4c94185-263e-4d4b-8772-19a7312f64b4_isrForCrosstalkSources_995991_12.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash af159f0dc5c2a2aee6c42b5eb954f342 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/51bb0312-5c23-4f33-a299-a9da16f1b5a1_isrForCrosstalkSources_995992_47.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/51bb0312-5c23-4f33-a299-a9da16f1b5a1_isrForCrosstalkSources_995992_47.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash f856cb977c02da89724ee4f1b83a7aac parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a9b99d6a-980d-4a82-a6dc-f39435fe7b14_isrForCrosstalkSources_995992_52.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a9b99d6a-980d-4a82-a6dc-f39435fe7b14_isrForCrosstalkSources_995992_52.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 0b6ac0c0925a281f5d7cb675c0c92910 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba914760-b173-4589-922e-1ac95e3ba2dc_isrForCrosstalkSources_995847_6.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba914760-b173-4589-922e-1ac95e3ba2dc_isrForCrosstalkSources_995847_6.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash f125e2672d6fc060b3405676f8000077 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/54a442df-1e41-4376-a223-3c1b14d12394_isrForCrosstalkSources_995992_56.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/54a442df-1e41-4376-a223-3c1b14d12394_isrForCrosstalkSources_995992_56.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 3034fef8b17041ca5885bdc8bacd0bca parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f21b252a-0b80-4e60-87ce-09f9f9bfac22_isrForCrosstalkSources_995845_5.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f21b252a-0b80-4e60-87ce-09f9f9bfac22_isrForCrosstalkSources_995845_5.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 4d127d8039b3bd38ac1684b18ebd1629 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2bb1ce69-6126-4791-bc0d-871b28417acd_isrForCrosstalkSources_995990_48.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2bb1ce69-6126-4791-bc0d-871b28417acd_isrForCrosstalkSources_995990_48.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash f1b51cba0696290da363ed342ac95b34 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/61f8c355-60a2-4d96-82e3-864664fcea9b_isrForCrosstalkSources_995992_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/61f8c355-60a2-4d96-82e3-864664fcea9b_isrForCrosstalkSources_995992_25.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 25f58cd93403930d6bf5e6bb9efe8499 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d7043dc-a790-4ab5-81cb-8ab666cfd9b0_isrForCrosstalkSources_995992_11.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d7043dc-a790-4ab5-81cb-8ab666cfd9b0_isrForCrosstalkSources_995992_11.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 2c9d90e76a9fea9d1e0809652e9c8554 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a740a5cb-c0d9-46cd-b93e-407efacad0da_isrForCrosstalkSources_995846_37.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a740a5cb-c0d9-46cd-b93e-407efacad0da_isrForCrosstalkSources_995846_37.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash fd10635ea611306358f8bf1fb8863710 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a16813b4-c791-4591-b879-64a38ffd2ac4_isrForCrosstalkSources_995847_36.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a16813b4-c791-4591-b879-64a38ffd2ac4_isrForCrosstalkSources_995847_36.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash c126a0ebed61f21b8a07a96a94208324 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/1ca87e60-7b7b-4b69-b7fa-9b355cef242b_isrForCrosstalkSources_995846_34.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/1ca87e60-7b7b-4b69-b7fa-9b355cef242b_isrForCrosstalkSources_995846_34.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash ec69f97d36cb69c02ec81eeaec09b124 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/94202a2c-1b56-4f9f-8400-6290a4ef8a61_isrForCrosstalkSources_995847_26.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/94202a2c-1b56-4f9f-8400-6290a4ef8a61_isrForCrosstalkSources_995847_26.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash c8ddd606fcc66e1fc6ffe7e6f7d7a897 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ccebc81f-c58c-4fc1-95b0-c11b2e7bc945_isrForCrosstalkSources_995992_21.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ccebc81f-c58c-4fc1-95b0-c11b2e7bc945_isrForCrosstalkSources_995992_21.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 629c5ace16e8267b668c13be57a6196b parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cde90d42-aa7c-472e-95f5-beadcb253dee_isrForCrosstalkSources_995845_38.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cde90d42-aa7c-472e-95f5-beadcb253dee_isrForCrosstalkSources_995845_38.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 2c08d4d18db9de7615eda6098aedd4f0 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3e52a3ee-7887-47d7-8d82-be0e9d47e8ce_isrForCrosstalkSources_995845_26.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3e52a3ee-7887-47d7-8d82-be0e9d47e8ce_isrForCrosstalkSources_995845_26.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash ef04189dcaad6243a8cd84135a6793ea parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4375044b-b4b0-4592-9065-d2b6901b6af8_isrForCrosstalkSources_995847_19.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4375044b-b4b0-4592-9065-d2b6901b6af8_isrForCrosstalkSources_995847_19.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash d74f1d2f8074f9e9d8abf725190fc19b parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cc0949d0-b847-4883-ad57-c213d5acfb4f_isrForCrosstalkSources_995845_50.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cc0949d0-b847-4883-ad57-c213d5acfb4f_isrForCrosstalkSources_995845_50.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 38809c140049a9e0844765a77b97fd2b parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8a0d037-b9cc-404e-8514-cd9c8d8a0cf7_isrForCrosstalkSources_995846_56.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8a0d037-b9cc-404e-8514-cd9c8d8a0cf7_isrForCrosstalkSources_995846_56.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 18214c07497f6b4be5666a034f39095b parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b4255a5e-277c-4b96-adf7-03fcbcfa32c0_isrForCrosstalkSources_995991_29.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b4255a5e-277c-4b96-adf7-03fcbcfa32c0_isrForCrosstalkSources_995991_29.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 332a1551b8e581f321287bdb5805100f parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/50cc8441-b41c-4fc2-8777-f07f61a01183_isrForCrosstalkSources_995991_37.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/50cc8441-b41c-4fc2-8777-f07f61a01183_isrForCrosstalkSources_995991_37.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 911635597f21054145a291b541180bee parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/7f8ff80b-9bd4-4db2-9241-a0c6517e71fc_isrForCrosstalkSources_995846_12.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/7f8ff80b-9bd4-4db2-9241-a0c6517e71fc_isrForCrosstalkSources_995846_12.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash ee3af5415f135e2f84ecde53045b3594 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/22ea96b8-cba3-4c81-93a7-2c2971387b9e_isrForCrosstalkSources_995992_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/22ea96b8-cba3-4c81-93a7-2c2971387b9e_isrForCrosstalkSources_995992_31.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 428c8660632bba91cdfb76d85457cfce parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7d32c36a-8567-464e-afa7-2691b130e52a_isrForCrosstalkSources_995991_43.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7d32c36a-8567-464e-afa7-2691b130e52a_isrForCrosstalkSources_995991_43.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash acce224739edbe0db6ef18bf77fc6d56 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/531b193b-e1b6-4189-a151-f0496d0a5b67_isrForCrosstalkSources_995845_1.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/531b193b-e1b6-4189-a151-f0496d0a5b67_isrForCrosstalkSources_995845_1.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 4ab80ce855ce9e930e400a4d179d34b6 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7bfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/626fe7a2-3677-4b6d-8836-20cca4392c44_isrForCrosstalkSources_995847_11.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/626fe7a2-3677-4b6d-8836-20cca4392c44_isrForCrosstalkSources_995847_11.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 9bd84f03ab62419ab5059c79885366bc parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ff1dc257-c9f8-4a96-b349-a654d20dfad7_isrForCrosstalkSources_995847_9.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ff1dc257-c9f8-4a96-b349-a654d20dfad7_isrForCrosstalkSources_995847_9.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 0d279f4dc6bb909c1400dbc6d83c5a41 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b6800e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/befd785b-c2b3-4d47-9bb0-2d2f4a694565_isrForCrosstalkSources_995992_43.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/befd785b-c2b3-4d47-9bb0-2d2f4a694565_isrForCrosstalkSources_995992_43.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash fb330cbf44d771466988991ac7b3faac parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b680180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7aa3aa16-1f97-436e-b5f1-afe82a834446_isrForCrosstalkSources_995992_38.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7aa3aa16-1f97-436e-b5f1-afe82a834446_isrForCrosstalkSources_995992_38.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash f21abdde8c80b8c36064a2ab7b9d78f1 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b6802c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 10:03:54 proc_lsst.multi:170 [INFO] [multi] job_id 17063567 proc_lsst.multi INFO: [multi] job_id 17063567 2024-03-15 10:03:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 10:03:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063567 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063567 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17063567 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-15 10:03:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-15 10:03:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 10:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69d81174-993b-4546-aa02-794ca3e90d8e_isrForCrosstalkSources_995846_17.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69d81174-993b-4546-aa02-794ca3e90d8e_isrForCrosstalkSources_995846_17.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 9e3018bacd4afd8ea767423de029e537 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/28f298ea-dd3e-4e41-a321-746fd2cbfdcc_isrForCrosstalkSources_995845_51.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/28f298ea-dd3e-4e41-a321-746fd2cbfdcc_isrForCrosstalkSources_995845_51.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 8ff78cf6ca73c79a083e493beb991b64 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b7be480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/924fb05d-4469-48f6-9177-aec025fce00e_isrForCrosstalkSources_995847_34.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/924fb05d-4469-48f6-9177-aec025fce00e_isrForCrosstalkSources_995847_34.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 04c9b5a892b4670542d7407be452d1cd parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b680400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6002d2dc-79a3-4775-abf4-be7c4ff4d423_isrForCrosstalkSources_995992_5.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6002d2dc-79a3-4775-abf4-be7c4ff4d423_isrForCrosstalkSources_995992_5.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 338a7ede98d0acd4594aa903316f8e15 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b6837e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ac719d69-5556-45f8-a87a-ec612c2057fc_isrForCrosstalkSources_995845_59.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ac719d69-5556-45f8-a87a-ec612c2057fc_isrForCrosstalkSources_995845_59.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash ba94d6b869b84189a3bd056c63dc32df parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b6804a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/6d0789ef-a2bd-4879-ac80-e3c2773f5c05_isrForCrosstalkSources_995847_7.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/6d0789ef-a2bd-4879-ac80-e3c2773f5c05_isrForCrosstalkSources_995847_7.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 4bd18825c3d19647b7eb60f80f7699f1 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b6836a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/33a03a2a-6ef4-425d-aef2-ab8fbc1a1cf9_isrForCrosstalkSources_995845_13.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/33a03a2a-6ef4-425d-aef2-ab8fbc1a1cf9_isrForCrosstalkSources_995845_13.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 30c5d279b43f1d136621fb9fb4df872b parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b680680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a1e45d64-5b78-4cfc-b956-a6027c31e7e7_isrForCrosstalkSources_995846_8.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a1e45d64-5b78-4cfc-b956-a6027c31e7e7_isrForCrosstalkSources_995846_8.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 754d254b1a6734db75e19701c3e29f27 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14e53b680720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/228d6d32-2cdf-4ece-b959-aef1b76f417b_isrForCrosstalkSources_995991_54.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/228d6d32-2cdf-4ece-b959-aef1b76f417b_isrForCrosstalkSources_995991_54.stderr 2024-03-15 10:03:54 proc_lsst.multi:170 [INFO] [multi] job_id 17063568 proc_lsst.multi INFO: [multi] job_id 17063568 2024-03-15 10:03:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 10:03:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063568 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063568 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17063568 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-15 10:03:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-15 10:03:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 10:03:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:55 proc_lsst.multi:170 [INFO] [multi] job_id 17063569 proc_lsst.multi INFO: [multi] job_id 17063569 2024-03-15 10:03:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 10:03:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063569 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063569 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17063569 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-15 10:03:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-15 10:03:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 10:03:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:55 proc_lsst.multi:170 [INFO] [multi] job_id 17063571 proc_lsst.multi INFO: [multi] job_id 17063571 2024-03-15 10:03:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 10:03:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063571 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063571 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17063571 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-15 10:03:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-15 10:03:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 10:03:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:56 proc_lsst.multi:170 [INFO] [multi] job_id 17063572 proc_lsst.multi INFO: [multi] job_id 17063572 2024-03-15 10:03:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 10:03:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063572 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063572 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17063572 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-15 10:03:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-15 10:03:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 10:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:56 proc_lsst.multi:170 [INFO] [multi] job_id 17063573 proc_lsst.multi INFO: [multi] job_id 17063573 2024-03-15 10:03:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 10:03:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063573 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063573 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17063573 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-15 10:03:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-15 10:03:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 10:03:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:57 proc_lsst.multi:170 [INFO] [multi] job_id 17063574 proc_lsst.multi INFO: [multi] job_id 17063574 2024-03-15 10:03:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 10:03:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063574 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063574 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17063574 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-15 10:03:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-15 10:03:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 10:03:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:58 proc_lsst.multi:170 [INFO] [multi] job_id 17063575 proc_lsst.multi INFO: [multi] job_id 17063575 2024-03-15 10:03:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 10:03:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063575 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063575 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17063575 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-15 10:03:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-15 10:03:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 10:03:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:59 proc_lsst.multi:170 [INFO] [multi] job_id 17063577 proc_lsst.multi INFO: [multi] job_id 17063577 2024-03-15 10:03:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 10:03:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063577 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063577 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17063577 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-15 10:03:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-15 10:03:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 10:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:03:59 proc_lsst.multi:170 [INFO] [multi] job_id 17063578 proc_lsst.multi INFO: [multi] job_id 17063578 2024-03-15 10:03:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 10:03:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063578 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063578 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17063578 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-15 10:03:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-15 10:03:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:03:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:03:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 10:03:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:03:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:00 proc_lsst.multi:170 [INFO] [multi] job_id 17063579 proc_lsst.multi INFO: [multi] job_id 17063579 2024-03-15 10:04:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 10:04:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063579 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063579 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17063579 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-15 10:04:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-15 10:04:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 10:04:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:00 proc_lsst.multi:170 [INFO] [multi] job_id 17063580 proc_lsst.multi INFO: [multi] job_id 17063580 2024-03-15 10:04:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 10:04:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063580 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063580 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17063580 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-15 10:04:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-15 10:04:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 10:04:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:01 proc_lsst.multi:170 [INFO] [multi] job_id 17063581 proc_lsst.multi INFO: [multi] job_id 17063581 2024-03-15 10:04:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 10:04:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063581 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063581 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17063581 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-15 10:04:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-15 10:04:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 10:04:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:02 proc_lsst.multi:170 [INFO] [multi] job_id 17063582 proc_lsst.multi INFO: [multi] job_id 17063582 2024-03-15 10:04:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 10:04:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063582 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063582 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17063582 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-15 10:04:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-15 10:04:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 10:04:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:03 proc_lsst.multi:170 [INFO] [multi] job_id 17063583 proc_lsst.multi INFO: [multi] job_id 17063583 2024-03-15 10:04:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 10:04:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063583 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063583 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17063583 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-15 10:04:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-15 10:04:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 10:04:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 882d852db76e1ff48783b9cceb3a66aa with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/35bf5d40-dba6-4368-a692-e4d4bbbdbcae_isrForCrosstalkSources_995992_39.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/35bf5d40-dba6-4368-a692-e4d4bbbdbcae_isrForCrosstalkSources_995992_39.stderr 2024-03-15 10:04:04 proc_lsst.multi:170 [INFO] [multi] job_id 17063584 proc_lsst.multi INFO: [multi] job_id 17063584 2024-03-15 10:04:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 10:04:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063584 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063584 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17063584 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-15 10:04:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-15 10:04:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 10:04:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 340293b9e2ea72378774257ae2ee8d9b with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d92f2779-73d2-4fd5-966e-f5df6e105921_isrForCrosstalkSources_995992_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d92f2779-73d2-4fd5-966e-f5df6e105921_isrForCrosstalkSources_995992_42.stderr 2024-03-15 10:04:05 proc_lsst.multi:170 [INFO] [multi] job_id 17063585 proc_lsst.multi INFO: [multi] job_id 17063585 2024-03-15 10:04:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 10:04:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063585 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063585 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17063585 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-15 10:04:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-15 10:04:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 10:04:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:06 proc_lsst.multi:170 [INFO] [multi] job_id 17063586 proc_lsst.multi INFO: [multi] job_id 17063586 2024-03-15 10:04:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 10:04:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063586 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063586 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17063586 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-15 10:04:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-15 10:04:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 10:04:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea65083dc4b609b7d4bc830c34719623 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5c935824-e56a-4821-adf0-69dc2543d6ec_isrForCrosstalkSources_995847_47.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5c935824-e56a-4821-adf0-69dc2543d6ec_isrForCrosstalkSources_995847_47.stderr 2024-03-15 10:04:09 proc_lsst.multi:170 [INFO] [multi] job_id 17063587 proc_lsst.multi INFO: [multi] job_id 17063587 2024-03-15 10:04:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 10:04:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063587 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063587 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17063587 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-15 10:04:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-15 10:04:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 10:04:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 10:04:10 proc_lsst.multi:170 [INFO] [multi] job_id 17063589 proc_lsst.multi INFO: [multi] job_id 17063589 2024-03-15 10:04:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 10:04:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063589 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063589 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17063589 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-15 10:04:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-15 10:04:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 10:04:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8cae11ced154b0214f3686a49af50c with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/835f257c-d8ac-4d8a-986f-4c231da08b9f_isrForCrosstalkSources_995991_15.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/835f257c-d8ac-4d8a-986f-4c231da08b9f_isrForCrosstalkSources_995991_15.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e8166402508d9c26e58cca563f594e4 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cd020327-ff4f-44a2-8df4-c026a8f5badd_isrForCrosstalkSources_995845_12.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cd020327-ff4f-44a2-8df4-c026a8f5badd_isrForCrosstalkSources_995845_12.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6371145eccaba370e7b54cc87e94961 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c4260cb5-4b20-41a8-a399-ea8cfed632a6_isrForCrosstalkSources_995991_42.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c4260cb5-4b20-41a8-a399-ea8cfed632a6_isrForCrosstalkSources_995991_42.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1477d573205724312a456b1d84a70ff5 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3db351f1-39b1-4c44-8da8-675a47d1c5f8_isrForCrosstalkSources_995845_48.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3db351f1-39b1-4c44-8da8-675a47d1c5f8_isrForCrosstalkSources_995845_48.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a521560dadbde1d453ad7f3abd89098 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d8aa8680-49a1-4cee-8e31-f3a12226a887_isrForCrosstalkSources_995991_56.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d8aa8680-49a1-4cee-8e31-f3a12226a887_isrForCrosstalkSources_995991_56.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897ef4eb2c86007864215bb4b2e67b0a with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/463ff919-0ded-43a0-aba5-31c4cc570b19_isrForCrosstalkSources_995991_59.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/463ff919-0ded-43a0-aba5-31c4cc570b19_isrForCrosstalkSources_995991_59.stderr 2024-03-15 10:04:12 proc_lsst.multi:170 [INFO] [multi] job_id 17063590 proc_lsst.multi INFO: [multi] job_id 17063590 2024-03-15 10:04:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 10:04:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063590 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063590 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17063590 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-15 10:04:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-15 10:04:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 10:04:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee1afe3e85c2774b3f5e29df0938a10e with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d7dda995-0445-4070-88a8-ae7e353ad480_isrForCrosstalkSources_995846_15.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d7dda995-0445-4070-88a8-ae7e353ad480_isrForCrosstalkSources_995846_15.stderr 2024-03-15 10:04:13 proc_lsst.multi:170 [INFO] [multi] job_id 17063591 proc_lsst.multi INFO: [multi] job_id 17063591 2024-03-15 10:04:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 10:04:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063591 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063591 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17063591 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-15 10:04:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-15 10:04:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 10:04:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e4d38a62b97c960a40ff8a296a44679 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fab65104-e10c-48ab-a38e-5bc4735cb2a1_isrForCrosstalkSources_995845_58.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fab65104-e10c-48ab-a38e-5bc4735cb2a1_isrForCrosstalkSources_995845_58.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edb7b91c0cf7191e5ab096210b224688 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/11937b21-5244-4962-8328-8f7b5747fd3b_isrForCrosstalkSources_995991_35.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/11937b21-5244-4962-8328-8f7b5747fd3b_isrForCrosstalkSources_995991_35.stderr 2024-03-15 10:04:15 proc_lsst.multi:170 [INFO] [multi] job_id 17063592 proc_lsst.multi INFO: [multi] job_id 17063592 2024-03-15 10:04:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 10:04:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063592 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063592 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17063592 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-15 10:04:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-15 10:04:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 10:04:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4f7d16699142e9879a21e8eeff3a8b9 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ea214338-3e35-4ea6-8718-757415e4ead3_isrForCrosstalkSources_995846_25.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ea214338-3e35-4ea6-8718-757415e4ead3_isrForCrosstalkSources_995846_25.stderr 2024-03-15 10:04:17 proc_lsst.multi:170 [INFO] [multi] job_id 17063593 proc_lsst.multi INFO: [multi] job_id 17063593 2024-03-15 10:04:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 10:04:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063593 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063593 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17063593 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-15 10:04:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-15 10:04:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 10:04:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa6fd0b77db0d7d51ba38d7d0d61e39 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/27285127-9683-470f-86d6-75f24b0fefbb_isrForCrosstalkSources_995991_60.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/27285127-9683-470f-86d6-75f24b0fefbb_isrForCrosstalkSources_995991_60.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ded2119edcb770e6616d53ad0a65179 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/90dd4cc7-6723-4b93-bad3-6ff383d60b99_isrForCrosstalkSources_995991_31.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/90dd4cc7-6723-4b93-bad3-6ff383d60b99_isrForCrosstalkSources_995991_31.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f0a79e00d0432e2ebd513b8052ffde with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ff6460f8-1fa4-4313-85b9-02782fa24783_isrForCrosstalkSources_995846_48.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/ff6460f8-1fa4-4313-85b9-02782fa24783_isrForCrosstalkSources_995846_48.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93e28ef2bcd3864c502298f2020a3291 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2536cfa3-9266-4098-b02c-ba2d6f72fb93_isrForCrosstalkSources_995991_22.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2536cfa3-9266-4098-b02c-ba2d6f72fb93_isrForCrosstalkSources_995991_22.stderr 2024-03-15 10:04:19 proc_lsst.multi:170 [INFO] [multi] job_id 17063594 proc_lsst.multi INFO: [multi] job_id 17063594 2024-03-15 10:04:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 10:04:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063594 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063594 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17063594 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-15 10:04:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-15 10:04:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 10:04:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c3840400dc29636a08a3debeac2b8a3 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/323ce1bd-eb67-46c6-bdfe-8f47c674a22a_isrForCrosstalkSources_995991_49.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/323ce1bd-eb67-46c6-bdfe-8f47c674a22a_isrForCrosstalkSources_995991_49.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3889d21b5809c2424e63a1c71552ba3 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5c633328-a323-4fc9-80c2-efa611f9c215_isrForCrosstalkSources_995846_11.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5c633328-a323-4fc9-80c2-efa611f9c215_isrForCrosstalkSources_995846_11.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8b32b7524cf86641d90ba6f5c81209 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df051932-a21a-4fd8-bfc3-1415e70c417b_isrForCrosstalkSources_995846_1.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df051932-a21a-4fd8-bfc3-1415e70c417b_isrForCrosstalkSources_995846_1.stderr 2024-03-15 10:04:21 proc_lsst.multi:170 [INFO] [multi] job_id 17063595 proc_lsst.multi INFO: [multi] job_id 17063595 2024-03-15 10:04:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17063595 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17063595 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17063595 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-15 10:04:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-15 10:04:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 10:04:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3622ab1ed8a9de53f8a57a059a1963a with result from task 18 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/41c0209d-30e7-4bfc-8adb-0e9bb9dae078_isrForCrosstalkSources_995991_36.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/41c0209d-30e7-4bfc-8adb-0e9bb9dae078_isrForCrosstalkSources_995991_36.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 536e7197dfc0c5870ace0f035484816a with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c991212b-1b80-4835-b814-3e8f2be3a6a0_isrForCrosstalkSources_995991_50.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c991212b-1b80-4835-b814-3e8f2be3a6a0_isrForCrosstalkSources_995991_50.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 406621d0f3cf7e6301c89eb617d065a6 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93b18cdc-9a59-4c90-b422-59c1a09ccbd0_isrForCrosstalkSources_995992_53.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93b18cdc-9a59-4c90-b422-59c1a09ccbd0_isrForCrosstalkSources_995992_53.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54126c7b107ff90738048cc1ef7e10af with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3d5fb016-f33c-4298-a70a-0e7ea7ba03be_isrForCrosstalkSources_995847_1.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3d5fb016-f33c-4298-a70a-0e7ea7ba03be_isrForCrosstalkSources_995847_1.stderr 2024-03-15 10:04:22 proc_lsst.multi:170 [INFO] [multi] job_id 17063596 proc_lsst.multi INFO: [multi] job_id 17063596 2024-03-15 10:04:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:22 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17063596 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17063596 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17063596 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-15 10:04:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-15 10:04:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c91c66c5e1a8e415cc25ca645732bb0f with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/20bec065-4fc3-4575-87de-e1bf16941cf4_isrForCrosstalkSources_995991_18.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/20bec065-4fc3-4575-87de-e1bf16941cf4_isrForCrosstalkSources_995991_18.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb2ca3e09a2cf7f5644618523c89654b with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/35094275-4daa-4ff3-bb79-95e46a6320f4_isrForCrosstalkSources_995847_14.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/35094275-4daa-4ff3-bb79-95e46a6320f4_isrForCrosstalkSources_995847_14.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23d652f82915634e014ee876204741dc with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/21817f50-7180-47e7-bb67-c4856ed9c96b_isrForCrosstalkSources_995846_42.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/21817f50-7180-47e7-bb67-c4856ed9c96b_isrForCrosstalkSources_995846_42.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8de61b723ebb561f3512555f08b59a79 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c417b168-0e47-49e8-a0a7-5cac45927923_isrForCrosstalkSources_995990_34.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c417b168-0e47-49e8-a0a7-5cac45927923_isrForCrosstalkSources_995990_34.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d70aea15ced0d5f600cb1d64f50e32c with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f08887bf-c684-4744-aa94-ca4c9e890e4b_isrForCrosstalkSources_995991_7.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f08887bf-c684-4744-aa94-ca4c9e890e4b_isrForCrosstalkSources_995991_7.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 212a3ad76bf17ec5d7e640628b5a8c99 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7cd437ab-7efa-48aa-8d84-a2bb1bbb8b33_isrForCrosstalkSources_995990_9.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7cd437ab-7efa-48aa-8d84-a2bb1bbb8b33_isrForCrosstalkSources_995990_9.stderr 2024-03-15 10:04:24 proc_lsst.multi:170 [INFO] [multi] job_id 17063597 proc_lsst.multi INFO: [multi] job_id 17063597 2024-03-15 10:04:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 10:04:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:24 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17063597 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17063597 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17063597 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-15 10:04:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-15 10:04:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 10:04:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb208db5c2c85bebf17d8c5867cd08be with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/76257e55-ddd6-47cd-b506-46ceada84227_isrForCrosstalkSources_995992_32.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/76257e55-ddd6-47cd-b506-46ceada84227_isrForCrosstalkSources_995992_32.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a697ebae7d74e0841aa987a9ea9d7ae with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/16914a28-d0e5-49e6-b38d-47af2a84618c_isrForCrosstalkSources_995845_56.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/16914a28-d0e5-49e6-b38d-47af2a84618c_isrForCrosstalkSources_995845_56.stderr 2024-03-15 10:04:26 proc_lsst.multi:170 [INFO] [multi] job_id 17063599 proc_lsst.multi INFO: [multi] job_id 17063599 2024-03-15 10:04:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 10:04:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:26 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17063599 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17063599 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17063599 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-15 10:04:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-15 10:04:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 10:04:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a23c05c4f774c041a9746588c0add75 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/b0668813-4351-473f-a349-5bf6a941c6ab_isrForCrosstalkSources_995845_61.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/b0668813-4351-473f-a349-5bf6a941c6ab_isrForCrosstalkSources_995845_61.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21081e7cf8bd072355b39d47fe1db7a2 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fa427cf6-e1dd-4357-b65b-76e63376a5ed_isrForCrosstalkSources_995992_2.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fa427cf6-e1dd-4357-b65b-76e63376a5ed_isrForCrosstalkSources_995992_2.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 633e904678d2ca7281581b482a565bf8 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2f75b453-387a-4809-99c7-1246eed11093_isrForCrosstalkSources_995846_58.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2f75b453-387a-4809-99c7-1246eed11093_isrForCrosstalkSources_995846_58.stderr 2024-03-15 10:04:28 proc_lsst.multi:170 [INFO] [multi] job_id 17063601 proc_lsst.multi INFO: [multi] job_id 17063601 2024-03-15 10:04:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 10:04:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:28 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17063601 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17063601 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17063601 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-15 10:04:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-15 10:04:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 10:04:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5e561f06dfd3fa369f389474c08c6f with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/cc8853a9-2983-4cb7-a46d-d5a0ebac42f8_isrForCrosstalkSources_995991_23.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/cc8853a9-2983-4cb7-a46d-d5a0ebac42f8_isrForCrosstalkSources_995991_23.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9190e16f00ae9b8e7d45d787770d4c8d with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/85812fdd-c377-4ea8-8585-8ba02f24a37b_isrForCrosstalkSources_995990_19.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/85812fdd-c377-4ea8-8585-8ba02f24a37b_isrForCrosstalkSources_995990_19.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2913f38f004be34d48c988074df8637b with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/661ae5b1-ebce-4719-b5e8-844d39a61806_isrForCrosstalkSources_995847_44.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/661ae5b1-ebce-4719-b5e8-844d39a61806_isrForCrosstalkSources_995847_44.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36349037e7ae106e0042032c84b47282 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c6bbb435-04ae-4631-96e9-d0ab0b7a9f36_isrForCrosstalkSources_995847_46.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c6bbb435-04ae-4631-96e9-d0ab0b7a9f36_isrForCrosstalkSources_995847_46.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eb9dc42ad0cd95ac1ee7b198e8d2231 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a329cae9-d65a-4b91-aa97-da15bdc5b440_isrForCrosstalkSources_995990_3.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a329cae9-d65a-4b91-aa97-da15bdc5b440_isrForCrosstalkSources_995990_3.stderr 2024-03-15 10:04:30 proc_lsst.multi:170 [INFO] [multi] job_id 17063602 proc_lsst.multi INFO: [multi] job_id 17063602 2024-03-15 10:04:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 10:04:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:30 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17063602 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17063602 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17063602 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-15 10:04:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-15 10:04:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 10:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 10:04:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 10:04:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f59e7f5ad2f49d207141b4b30d35c3d4 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e3a27af-3543-4606-b2cb-9fc14ed868bd_isrForCrosstalkSources_995992_34.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e3a27af-3543-4606-b2cb-9fc14ed868bd_isrForCrosstalkSources_995992_34.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8620bf09692e45f8227bb8d55befaa with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/04bf41da-fb5f-4c46-96ff-16a1b8d516c4_isrForCrosstalkSources_995847_53.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/04bf41da-fb5f-4c46-96ff-16a1b8d516c4_isrForCrosstalkSources_995847_53.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 557b68d0f8aca8d261e0974d675d7a79 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/937c8b74-5dd1-494f-bc01-105f043f1e61_isrForCrosstalkSources_995846_39.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/937c8b74-5dd1-494f-bc01-105f043f1e61_isrForCrosstalkSources_995846_39.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c24b09c1e33d89ba8ce1702126a5223e with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/da315e9d-90d7-4560-a47c-1a0adfef95ca_isrForCrosstalkSources_995846_14.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/da315e9d-90d7-4560-a47c-1a0adfef95ca_isrForCrosstalkSources_995846_14.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97105c9052ed99d779bdb62f43d715a3 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f5da9a35-2cbc-4558-bbe3-210434c7161a_isrForCrosstalkSources_995845_14.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f5da9a35-2cbc-4558-bbe3-210434c7161a_isrForCrosstalkSources_995845_14.stderr 2024-03-15 10:04:32 proc_lsst.multi:170 [INFO] [multi] job_id 17063603 proc_lsst.multi INFO: [multi] job_id 17063603 2024-03-15 10:04:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 10:04:32 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17063603 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17063603 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17063603 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-15 10:04:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-15 10:04:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 10:04:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 10:04:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 10:04:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 10:04:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 10:04:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 10:04:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 10:04:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 10:04:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 10:04:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.65.9,10.64.129.9,169.254.95.120,127.0.0.1,198.48.92.26,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54801 --result_port=54411 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ca2bf230b0a48e9217f6611a7739eff with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/30777447-e809-4cfa-a076-5757ca018936_isrForCrosstalkSources_995990_31.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/30777447-e809-4cfa-a076-5757ca018936_isrForCrosstalkSources_995990_31.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db3cf2f48930c2da09b182e69a698d0 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7cbf917-3638-47d6-b0da-3af6294c8231_isrForCrosstalkSources_995992_33.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7cbf917-3638-47d6-b0da-3af6294c8231_isrForCrosstalkSources_995992_33.stderr 2024-03-15 10:04:34 proc_lsst.multi:170 [INFO] [multi] job_id 17063605 proc_lsst.multi INFO: [multi] job_id 17063605 2024-03-15 10:04:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 10:04:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 10:04:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17063605 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17063605 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17063605 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22974949607440 2024-03-15 10:04:34 proc_lsst.multi:146 [INFO] found job 56858 in provider local proc_lsst.multi INFO: found job 56858 in provider local parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a00253860774208a3ea39a6cc149c57a with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c202bfe6-b2bf-448a-856d-9fd244bf249b_isrForCrosstalkSources_995990_25.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c202bfe6-b2bf-448a-856d-9fd244bf249b_isrForCrosstalkSources_995990_25.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb60e4f1c5522493a241fc35eaf631ed with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fe4624ea-529b-4161-8998-17917dbd35d6_isrForCrosstalkSources_995992_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fe4624ea-529b-4161-8998-17917dbd35d6_isrForCrosstalkSources_995992_27.stderr 2024-03-15 10:04:34 proc_lsst.multi:146 [INFO] found job 17063563 in provider astro proc_lsst.multi INFO: found job 17063563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ed6f3a9953bea297e97b95695877d2 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3b85802a-2f08-48cc-a065-2c518f3d62ab_isrForCrosstalkSources_995990_55.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3b85802a-2f08-48cc-a065-2c518f3d62ab_isrForCrosstalkSources_995990_55.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f8ca953d3c4a333e94303160c608d06 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d569e4a4-1f81-43c5-be03-8a05b32292bb_isrForCrosstalkSources_995992_54.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d569e4a4-1f81-43c5-be03-8a05b32292bb_isrForCrosstalkSources_995992_54.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30a5c64745fce5afaed6b8b044b30efd with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b9540e2c-2ca0-4718-97fc-10eaedda582e_isrForCrosstalkSources_995991_55.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b9540e2c-2ca0-4718-97fc-10eaedda582e_isrForCrosstalkSources_995991_55.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 153a61f1563891f798f68d9960183ed6 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/336e0a8d-045b-42d2-adc1-8b315077ca82_isrForCrosstalkSources_995845_8.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/336e0a8d-045b-42d2-adc1-8b315077ca82_isrForCrosstalkSources_995845_8.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f949fa409d31a1973bac2c397fa21147 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ee7deebf-64d0-4c3a-99ca-85dbc8afd71d_isrForCrosstalkSources_995845_33.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ee7deebf-64d0-4c3a-99ca-85dbc8afd71d_isrForCrosstalkSources_995845_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:36 proc_lsst.multi:146 [INFO] found job 17063564 in provider astro proc_lsst.multi INFO: found job 17063564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cea83891f1c1ba57a4bebf975d4a985 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b449a9ad-e782-4f3c-8e39-712e09b8672b_isrForCrosstalkSources_995992_17.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b449a9ad-e782-4f3c-8e39-712e09b8672b_isrForCrosstalkSources_995992_17.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86de5156a2a261490ca9a7b56f5226fd with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c6fb5633-4d11-4713-965c-4affc39f2066_isrForCrosstalkSources_995992_40.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c6fb5633-4d11-4713-965c-4affc39f2066_isrForCrosstalkSources_995992_40.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67dd268e0dfc38b026280e65f650043c with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/6079bfd7-5b56-4d62-a726-4b1bab683e14_isrForCrosstalkSources_995990_39.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/6079bfd7-5b56-4d62-a726-4b1bab683e14_isrForCrosstalkSources_995990_39.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 602189faf01df3561181394e4c1b7467 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9400445c-d5b1-48b8-8d2c-c39df1ea3347_isrForCrosstalkSources_995845_36.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9400445c-d5b1-48b8-8d2c-c39df1ea3347_isrForCrosstalkSources_995845_36.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d266a1dfa7c8174ca0aeab706706da50 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5e2ac03d-4116-4826-85b0-e11db26ed7e6_isrForCrosstalkSources_995847_56.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5e2ac03d-4116-4826-85b0-e11db26ed7e6_isrForCrosstalkSources_995847_56.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f74d4896a8b174414e6d7b829ede85a with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cf90d5f0-9d3e-4f3c-97d9-90c636b57f2c_isrForCrosstalkSources_995845_24.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cf90d5f0-9d3e-4f3c-97d9-90c636b57f2c_isrForCrosstalkSources_995845_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:38 proc_lsst.multi:146 [INFO] found job 17063565 in provider astro proc_lsst.multi INFO: found job 17063565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4134f114c697cb86697c5b6bc9f78f42 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/bc994bfb-bfc7-41f6-ad70-122d3c28b96b_isrForCrosstalkSources_995847_35.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/bc994bfb-bfc7-41f6-ad70-122d3c28b96b_isrForCrosstalkSources_995847_35.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ee0cbae18ff32b3fa49ecf700ef4cc with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/65ed42de-02a8-47ae-bba3-9e8207c83bc0_isrForCrosstalkSources_995846_28.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/65ed42de-02a8-47ae-bba3-9e8207c83bc0_isrForCrosstalkSources_995846_28.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c07c6422883f77eb52edd07c8982c2b with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/59b0ce3e-7aac-43af-9c8b-1fae824e0bdb_isrForCrosstalkSources_995845_11.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/59b0ce3e-7aac-43af-9c8b-1fae824e0bdb_isrForCrosstalkSources_995845_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:40 proc_lsst.multi:146 [INFO] found job 17063566 in provider astro proc_lsst.multi INFO: found job 17063566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3a85cdee6cfd5cff2d6e8cdb48c3037 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c3aead3e-748a-458e-a282-564ea3da94d6_isrForCrosstalkSources_995991_53.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c3aead3e-748a-458e-a282-564ea3da94d6_isrForCrosstalkSources_995991_53.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 252539e6fc685efe1d724db52e3e33c4 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/336f8516-2848-43b8-aac4-24bfc6602ba1_isrForCrosstalkSources_995992_9.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/336f8516-2848-43b8-aac4-24bfc6602ba1_isrForCrosstalkSources_995992_9.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8cc566483658e952f5c34d84b7c35f6 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f06aea87-3b5f-40e8-ba80-606981ff3200_isrForCrosstalkSources_995991_5.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f06aea87-3b5f-40e8-ba80-606981ff3200_isrForCrosstalkSources_995991_5.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 545b5d79b80a71b67297efd94fde0e9e with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e66a9c1-e28f-435e-a01e-c2c7b885d67f_isrForCrosstalkSources_995992_20.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3e66a9c1-e28f-435e-a01e-c2c7b885d67f_isrForCrosstalkSources_995992_20.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c954e7c8f44f83039e79e2ab6235774 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1c5a936a-0609-4f29-9d0b-c0d0c882500b_isrForCrosstalkSources_995847_59.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1c5a936a-0609-4f29-9d0b-c0d0c882500b_isrForCrosstalkSources_995847_59.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b415eca1dcd16f4b1f27fbca51d21f13 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/721f5177-a9ba-40d5-9ff4-d7f013dfd2a8_isrForCrosstalkSources_995992_8.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/721f5177-a9ba-40d5-9ff4-d7f013dfd2a8_isrForCrosstalkSources_995992_8.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cf2e55c40fd5d521422395e27941782 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91d1b138-8375-41e2-985f-a55f3b0d4a49_isrForCrosstalkSources_995992_49.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91d1b138-8375-41e2-985f-a55f3b0d4a49_isrForCrosstalkSources_995992_49.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb1083a83f1cd825b3d252923d1d2e07 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c92bb445-f3d0-44d6-8e2a-f52c63604aa5_isrForCrosstalkSources_995845_22.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c92bb445-f3d0-44d6-8e2a-f52c63604aa5_isrForCrosstalkSources_995845_22.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dbe24088e495ee57c1e34d4a975081b with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/8f101ac4-bc65-40b5-967f-4a12b541e3b5_isrForCrosstalkSources_995845_52.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/8f101ac4-bc65-40b5-967f-4a12b541e3b5_isrForCrosstalkSources_995845_52.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97aebd40472502adca7af48f8dffd297 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e1b63eb7-7646-4222-b44f-13a05dc86041_isrForCrosstalkSources_995992_6.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e1b63eb7-7646-4222-b44f-13a05dc86041_isrForCrosstalkSources_995992_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:42 proc_lsst.multi:146 [INFO] found job 17063567 in provider astro proc_lsst.multi INFO: found job 17063567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a46824d84d9b15cb58926e5ff9f1112f with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2dc2233b-1f38-44ad-8e50-fe5628de7432_isrForCrosstalkSources_995990_14.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2dc2233b-1f38-44ad-8e50-fe5628de7432_isrForCrosstalkSources_995990_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:44 proc_lsst.multi:146 [INFO] found job 17063568 in provider astro proc_lsst.multi INFO: found job 17063568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ab670539e453880f6691b4333502cd5 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1af821d6-06a2-424f-a4ee-bd076ddeff34_isrForCrosstalkSources_995845_27.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1af821d6-06a2-424f-a4ee-bd076ddeff34_isrForCrosstalkSources_995845_27.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0d81f9179a4b87e94379e744b9fb439 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5b526511-bfd2-404b-a536-ee1f646df749_isrForCrosstalkSources_995847_28.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/5b526511-bfd2-404b-a536-ee1f646df749_isrForCrosstalkSources_995847_28.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7026260eb545041d5adc9a263b1df329 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/20b3e70d-c965-49a0-83d8-c14ea52a3caf_isrForCrosstalkSources_995846_29.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/20b3e70d-c965-49a0-83d8-c14ea52a3caf_isrForCrosstalkSources_995846_29.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9e304b3a996010b19f23ca10ceee88d with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/629e6089-9d0a-479f-8819-486bfefb148b_isrForCrosstalkSources_995991_32.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/629e6089-9d0a-479f-8819-486bfefb148b_isrForCrosstalkSources_995991_32.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdbd3d9afbcfa5f59001667fbdff760d with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07a47b58-89a7-48b5-ad83-08a6f751ec14_isrForCrosstalkSources_995845_60.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07a47b58-89a7-48b5-ad83-08a6f751ec14_isrForCrosstalkSources_995845_60.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31141b045fbd19bbeac6d828a91fd12e with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/92c67fa7-2ca4-45fb-acc8-c734c20541a0_isrForCrosstalkSources_995846_23.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/92c67fa7-2ca4-45fb-acc8-c734c20541a0_isrForCrosstalkSources_995846_23.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7b33ff59ea9824d395f7468f9fa1ce with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/55e10068-0527-44eb-adef-2224ecbb5253_isrForCrosstalkSources_995845_6.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/55e10068-0527-44eb-adef-2224ecbb5253_isrForCrosstalkSources_995845_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:46 proc_lsst.multi:146 [INFO] found job 17063569 in provider astro proc_lsst.multi INFO: found job 17063569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5276cbba9e7d82ae09886c8bb1fa053e with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8748009d-99f2-4a2e-8a4a-73bf8f4ea840_isrForCrosstalkSources_995847_49.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8748009d-99f2-4a2e-8a4a-73bf8f4ea840_isrForCrosstalkSources_995847_49.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6db6fada6aa628b56a1799ddf5a68257 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1f9cae6d-8f40-4d66-af8e-ecce2d254e98_isrForCrosstalkSources_995990_30.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1f9cae6d-8f40-4d66-af8e-ecce2d254e98_isrForCrosstalkSources_995990_30.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd2196bbbc0aba09ee10cad59d93adc4 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/282dd52a-9b83-4b30-8a07-dd556e5fa04a_isrForCrosstalkSources_995847_27.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/282dd52a-9b83-4b30-8a07-dd556e5fa04a_isrForCrosstalkSources_995847_27.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f1d2bb534bc41cd7369ae59b9fec0b6 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bee75ebe-37e3-4593-b856-57da5d7e6d13_isrForCrosstalkSources_995990_22.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bee75ebe-37e3-4593-b856-57da5d7e6d13_isrForCrosstalkSources_995990_22.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c90ab20fe658d3a82fe471e85e7549f1 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d9830d87-5ed6-4dc6-a511-cc7f8aa43b94_isrForCrosstalkSources_995991_28.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/d9830d87-5ed6-4dc6-a511-cc7f8aa43b94_isrForCrosstalkSources_995991_28.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ccfc7be6eed18a7fd6a0f7b87a97c5 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2aa15bec-d236-4fc8-9c81-012d1e6083ad_isrForCrosstalkSources_995990_18.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2aa15bec-d236-4fc8-9c81-012d1e6083ad_isrForCrosstalkSources_995990_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:48 proc_lsst.multi:146 [INFO] found job 17063571 in provider astro proc_lsst.multi INFO: found job 17063571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e2395565ccf1583a8d1040c69ef11de with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/565d0b86-fcca-4f4f-800a-c4cda4e2b766_isrForCrosstalkSources_995991_24.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/565d0b86-fcca-4f4f-800a-c4cda4e2b766_isrForCrosstalkSources_995991_24.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d81c788a8e5c14b0746d74f69a014a7 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/ee28cb78-a6b8-48f4-a910-4cf15c1231fd_isrForCrosstalkSources_995991_3.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/ee28cb78-a6b8-48f4-a910-4cf15c1231fd_isrForCrosstalkSources_995991_3.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5111c340e478469116ee3610287624b7 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/e2464fcd-30a1-458b-aaf2-253298a4c56d_isrForCrosstalkSources_995990_33.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/e2464fcd-30a1-458b-aaf2-253298a4c56d_isrForCrosstalkSources_995990_33.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 843381bc86bd7b591cc9cf4973bc8a24 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6a7d9a5c-eb6c-413e-ad5a-694b6c7d17d2_isrForCrosstalkSources_995846_13.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6a7d9a5c-eb6c-413e-ad5a-694b6c7d17d2_isrForCrosstalkSources_995846_13.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 779b9fb3106903a22ee38a3bb12cd76c with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2e326dde-9d44-4994-8775-6fb0253a8554_isrForCrosstalkSources_995846_4.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2e326dde-9d44-4994-8775-6fb0253a8554_isrForCrosstalkSources_995846_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:50 proc_lsst.multi:146 [INFO] found job 17063572 in provider astro proc_lsst.multi INFO: found job 17063572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca94f3575a128ed8a4e9e2f94566ad24 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b3457f0b-4ecc-48f1-81ba-b6f3086ee24e_isrForCrosstalkSources_995847_21.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b3457f0b-4ecc-48f1-81ba-b6f3086ee24e_isrForCrosstalkSources_995847_21.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec6a24b233869b2c2527fbcbd3c28026 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f13141f4-1f67-4ac3-81d8-062e637033af_isrForCrosstalkSources_995845_54.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f13141f4-1f67-4ac3-81d8-062e637033af_isrForCrosstalkSources_995845_54.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1683dd1a42958d9d4ada67f80c36d58d with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/da0526a1-4849-42c1-9f88-ac53aac773d2_isrForCrosstalkSources_995991_6.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/da0526a1-4849-42c1-9f88-ac53aac773d2_isrForCrosstalkSources_995991_6.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e249959d243f68d875f056256004caab with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/307189b1-d742-4dde-a92a-5431248ff31b_isrForCrosstalkSources_995845_25.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/307189b1-d742-4dde-a92a-5431248ff31b_isrForCrosstalkSources_995845_25.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57ff4d2650d42638e78eee9da646efdc with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/47399bfd-3ab3-4169-ab0e-dc975a7ce6b2_isrForCrosstalkSources_995845_44.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/47399bfd-3ab3-4169-ab0e-dc975a7ce6b2_isrForCrosstalkSources_995845_44.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce20e256dfa4c958c42cd95af5a1f075 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/846a4b2b-bd0b-4b91-ad5b-8d52a1cb3575_isrForCrosstalkSources_995990_35.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/846a4b2b-bd0b-4b91-ad5b-8d52a1cb3575_isrForCrosstalkSources_995990_35.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 132aac769c3b21ea505a8199ff93ea2f with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/39022dac-10ee-4f1e-ae53-5ea2f9f40119_isrForCrosstalkSources_995990_24.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/39022dac-10ee-4f1e-ae53-5ea2f9f40119_isrForCrosstalkSources_995990_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:52 proc_lsst.multi:146 [INFO] found job 17063573 in provider astro proc_lsst.multi INFO: found job 17063573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7e882cf5af3db00feee87cfcaf02b4d with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/71237ccb-1659-40fc-872d-c020ffbd4a2d_isrForCrosstalkSources_995846_41.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/71237ccb-1659-40fc-872d-c020ffbd4a2d_isrForCrosstalkSources_995846_41.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 857c65918f1d4a1582864ef6c4cf6bb7 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/f07a590e-90a6-41d2-80ce-1013ff488b63_isrForCrosstalkSources_995846_2.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/f07a590e-90a6-41d2-80ce-1013ff488b63_isrForCrosstalkSources_995846_2.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40eceb0a972459b2ea5a88bc9092dd15 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/c628900d-4f3b-4669-9b08-ee4fc466e6b5_isrForCrosstalkSources_995846_36.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/c628900d-4f3b-4669-9b08-ee4fc466e6b5_isrForCrosstalkSources_995846_36.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd6fa9ca83e4040e36a8a2e9a0fe8585 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/81f3f8ec-9981-4e89-831e-e26a6b97b5b3_isrForCrosstalkSources_995846_52.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/81f3f8ec-9981-4e89-831e-e26a6b97b5b3_isrForCrosstalkSources_995846_52.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46bffe20274bdf7505d697a6301ba296 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d3c0706-9594-430a-93bc-6f6c36b261ce_isrForCrosstalkSources_995846_62.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d3c0706-9594-430a-93bc-6f6c36b261ce_isrForCrosstalkSources_995846_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:54 proc_lsst.multi:146 [INFO] found job 17063574 in provider astro proc_lsst.multi INFO: found job 17063574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f3338bb50d409cac28c37c834682c28 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a620af50-aea3-4184-b623-f58d79bc0dec_isrForCrosstalkSources_995847_15.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a620af50-aea3-4184-b623-f58d79bc0dec_isrForCrosstalkSources_995847_15.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b7d5477b28d4d67f193df3a79f5f74f with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3ed1ae41-dcd3-4987-951c-4205fc6105ff_isrForCrosstalkSources_995846_7.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3ed1ae41-dcd3-4987-951c-4205fc6105ff_isrForCrosstalkSources_995846_7.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ac4a2e43c024b6f5f47d01f616ce0e7 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5543756f-4617-4fd5-8e50-04428b550382_isrForCrosstalkSources_995845_35.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5543756f-4617-4fd5-8e50-04428b550382_isrForCrosstalkSources_995845_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:56 proc_lsst.multi:146 [INFO] found job 17063575 in provider astro proc_lsst.multi INFO: found job 17063575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 466f76d9e8e8577fef5b3751d29f2bb8 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2b7294da-3a8b-400c-be9f-6454789a21df_isrForCrosstalkSources_995992_26.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2b7294da-3a8b-400c-be9f-6454789a21df_isrForCrosstalkSources_995992_26.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 046a6111a412067c2cb2b87ee9dd23bd with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/92bc9d7a-d2af-4158-883e-abffd7008768_isrForCrosstalkSources_995845_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/92bc9d7a-d2af-4158-883e-abffd7008768_isrForCrosstalkSources_995845_2.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff234997fa665351bbc7757f545945c4 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/46a636f5-4c6f-4348-a1a4-3492310107ea_isrForCrosstalkSources_995992_23.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/46a636f5-4c6f-4348-a1a4-3492310107ea_isrForCrosstalkSources_995992_23.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfed459d3a2c2dce788781175bfd461 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3c7b42c1-e5ad-4f63-b92a-a5eec5339727_isrForCrosstalkSources_995845_43.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3c7b42c1-e5ad-4f63-b92a-a5eec5339727_isrForCrosstalkSources_995845_43.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef1371d50c17f61e53391e791485abf4 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1ab3376-936a-4f47-b419-dd0cf19b2ffe_isrForCrosstalkSources_995846_5.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1ab3376-936a-4f47-b419-dd0cf19b2ffe_isrForCrosstalkSources_995846_5.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06af8e2908d9fea32c528fdfea1bd20 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c624229a-ba25-480d-8bd2-316f425f09b7_isrForCrosstalkSources_995992_29.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/c624229a-ba25-480d-8bd2-316f425f09b7_isrForCrosstalkSources_995992_29.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ef0e3dc7730f100c8ee1ee231f16388 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8e373c81-1296-4624-941d-2b13e5010119_isrForCrosstalkSources_995847_17.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8e373c81-1296-4624-941d-2b13e5010119_isrForCrosstalkSources_995847_17.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28f658afb2e3551a4a6078707e1f238b with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/50d3c9eb-40c4-45a0-a3ef-1fe07378555a_isrForCrosstalkSources_995846_44.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/50d3c9eb-40c4-45a0-a3ef-1fe07378555a_isrForCrosstalkSources_995846_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:04:58 proc_lsst.multi:146 [INFO] found job 17063577 in provider astro proc_lsst.multi INFO: found job 17063577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07ec2bf1c2ff934ccb8684470637cf72 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9858fe82-bf39-4f46-879f-ef7401e6481a_isrForCrosstalkSources_995845_18.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9858fe82-bf39-4f46-879f-ef7401e6481a_isrForCrosstalkSources_995845_18.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c4e8e57b361829bbdc7c3885cf36740 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/e1e5175d-db11-40f9-baea-93456d07e273_isrForCrosstalkSources_995845_9.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/e1e5175d-db11-40f9-baea-93456d07e273_isrForCrosstalkSources_995845_9.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e8e3fe5cab63b11a6bd9cda2c6705ce with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/5d213017-a99d-4531-857f-d265a4f807e6_isrForCrosstalkSources_995992_1.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/5d213017-a99d-4531-857f-d265a4f807e6_isrForCrosstalkSources_995992_1.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92db638c828fa17a471fba33de31070b with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/84e2f9dd-a326-418d-a169-e3dc4c87adf8_isrForCrosstalkSources_995992_16.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/84e2f9dd-a326-418d-a169-e3dc4c87adf8_isrForCrosstalkSources_995992_16.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0976a6d4f0c4ec80e0142c6b40aac6a7 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/eb24f8cc-d520-47cf-b228-bd85ef0f8ac8_isrForCrosstalkSources_995990_17.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/eb24f8cc-d520-47cf-b228-bd85ef0f8ac8_isrForCrosstalkSources_995990_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:00 proc_lsst.multi:146 [INFO] found job 17063578 in provider astro proc_lsst.multi INFO: found job 17063578 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4043987bd4eafb1297aee55003b07e with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c6be25ee-108b-4c61-92f6-2f79d9b4a749_isrForCrosstalkSources_995845_39.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/c6be25ee-108b-4c61-92f6-2f79d9b4a749_isrForCrosstalkSources_995845_39.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65b5fa39e4f2ae33403a6d9d95cc1be8 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9a741536-27b7-4eb3-903d-e40d72c35cc7_isrForCrosstalkSources_995845_29.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/9a741536-27b7-4eb3-903d-e40d72c35cc7_isrForCrosstalkSources_995845_29.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6144bb5dfbeb3bd9ff080e68c42d2a7 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/790bfe18-1b9c-47fb-a2e6-10436247ea90_isrForCrosstalkSources_995990_8.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/790bfe18-1b9c-47fb-a2e6-10436247ea90_isrForCrosstalkSources_995990_8.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b03ec063489064b1247808954290a92 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1c76213d-8c37-4256-8cdf-bfc81786afc2_isrForCrosstalkSources_995990_59.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1c76213d-8c37-4256-8cdf-bfc81786afc2_isrForCrosstalkSources_995990_59.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cb60cd05237a8a6dd6b3402e24158b with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/278c6923-99f6-4d1a-be99-e1e2315439e3_isrForCrosstalkSources_995845_17.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/278c6923-99f6-4d1a-be99-e1e2315439e3_isrForCrosstalkSources_995845_17.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e33e16e5e5383ebdd34ef883576a4a76 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fbfd1665-d593-4cef-9f68-ab8b9b9e8308_isrForCrosstalkSources_995990_7.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fbfd1665-d593-4cef-9f68-ab8b9b9e8308_isrForCrosstalkSources_995990_7.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc0c6db63465c4ebf60cdc465f9492f with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/fb7b142d-6abc-4c30-9915-f6940ebc30d7_isrForCrosstalkSources_995847_38.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/fb7b142d-6abc-4c30-9915-f6940ebc30d7_isrForCrosstalkSources_995847_38.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14988f8df722c2ac08a52f496cdad72a with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/387760f1-cf1e-4959-aa57-a1df7df4fff8_isrForCrosstalkSources_995990_10.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/387760f1-cf1e-4959-aa57-a1df7df4fff8_isrForCrosstalkSources_995990_10.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc98bf161c8eba858abfced12abd9b79 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ff7bdd17-9f3e-4012-874f-6752d7c6393b_isrForCrosstalkSources_995845_16.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ff7bdd17-9f3e-4012-874f-6752d7c6393b_isrForCrosstalkSources_995845_16.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61de25258ed28aecd748681bebd66284 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e357c3ef-472a-49ae-8d5a-bab668983c61_isrForCrosstalkSources_995991_10.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e357c3ef-472a-49ae-8d5a-bab668983c61_isrForCrosstalkSources_995991_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:02 proc_lsst.multi:146 [INFO] found job 17063579 in provider astro proc_lsst.multi INFO: found job 17063579 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f2d3522d107ebec72eb0d76761b28b6 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/af9b46c2-5e0f-4a68-8fbc-80436ee9d728_isrForCrosstalkSources_995991_44.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/af9b46c2-5e0f-4a68-8fbc-80436ee9d728_isrForCrosstalkSources_995991_44.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09edbd5ada553e35b230c356fd6c8981 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/72c6f4e0-323d-4dbc-a415-549de9f2c4df_isrForCrosstalkSources_995990_38.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/72c6f4e0-323d-4dbc-a415-549de9f2c4df_isrForCrosstalkSources_995990_38.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b68bf11cec0607e7432dbc0b3eaefeab with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/82215ea4-f02a-4ef1-873a-baf98dbd0b2f_isrForCrosstalkSources_995845_45.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/82215ea4-f02a-4ef1-873a-baf98dbd0b2f_isrForCrosstalkSources_995845_45.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e1c31a6563b18253ebef4e300e5e16 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8d066233-e5fa-460e-8e30-14ae4387722e_isrForCrosstalkSources_995847_20.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8d066233-e5fa-460e-8e30-14ae4387722e_isrForCrosstalkSources_995847_20.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38db01c48bfbd34e6811c5317a647aac with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b00818bd-80a4-4cd0-97c6-aa54eb3abc25_isrForCrosstalkSources_995991_20.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b00818bd-80a4-4cd0-97c6-aa54eb3abc25_isrForCrosstalkSources_995991_20.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6617bdb7f1965c6f59b94efdb25da6bf with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/81242608-706c-40e1-a38e-3cf08bf6f3ba_isrForCrosstalkSources_995991_27.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/81242608-706c-40e1-a38e-3cf08bf6f3ba_isrForCrosstalkSources_995991_27.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1214b9c47ce17924e0084a6f6e30ced with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b0ac35a2-30f8-4230-a35e-fdae59dbd931_isrForCrosstalkSources_995992_10.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b0ac35a2-30f8-4230-a35e-fdae59dbd931_isrForCrosstalkSources_995992_10.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b35f971a6bc8cc654466798b946e499d with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/876b9323-8193-4490-b6b6-f0a5416c8647_isrForCrosstalkSources_995846_9.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/876b9323-8193-4490-b6b6-f0a5416c8647_isrForCrosstalkSources_995846_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:04 proc_lsst.multi:146 [INFO] found job 17063580 in provider astro proc_lsst.multi INFO: found job 17063580 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68e9f4721b7165278c47ffedb97a79db with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5e2e4382-774b-426c-818a-de63e2c18794_isrForCrosstalkSources_995845_3.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5e2e4382-774b-426c-818a-de63e2c18794_isrForCrosstalkSources_995845_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:06 proc_lsst.multi:146 [INFO] found job 17063581 in provider astro proc_lsst.multi INFO: found job 17063581 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26b71d7ac394c25cfe4f4e109439baf9 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/344b02b0-d64c-4649-a77e-670fbcf968bc_isrForCrosstalkSources_995846_22.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/344b02b0-d64c-4649-a77e-670fbcf968bc_isrForCrosstalkSources_995846_22.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a963e2835d89428607622d83ccf4987 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c5839727-714f-4608-b478-1407897242a7_isrForCrosstalkSources_995990_32.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c5839727-714f-4608-b478-1407897242a7_isrForCrosstalkSources_995990_32.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11067d649ef858883a3bc631fe97b3a4 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87d305fc-b60f-4f8e-b4c6-b50a18a39e8f_isrForCrosstalkSources_995847_4.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87d305fc-b60f-4f8e-b4c6-b50a18a39e8f_isrForCrosstalkSources_995847_4.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbc66f7e3aff42daa182d2828f179bb9 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/d64bda82-f05d-4e3f-9de5-898ceab3d0e6_isrForCrosstalkSources_995847_54.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/d64bda82-f05d-4e3f-9de5-898ceab3d0e6_isrForCrosstalkSources_995847_54.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3582b94b50d37e46f3b939dd9af14092 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d9d9fc3c-d301-4e24-bfff-e88853c471d2_isrForCrosstalkSources_995846_26.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d9d9fc3c-d301-4e24-bfff-e88853c471d2_isrForCrosstalkSources_995846_26.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379c3415b11625fea610657ffbc2f38e with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbc48ee2-f4b0-467e-a5be-15697bfad714_isrForCrosstalkSources_995990_11.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbc48ee2-f4b0-467e-a5be-15697bfad714_isrForCrosstalkSources_995990_11.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c84ffbe5be4f2397d94c309d43ff42cb with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a6e8572c-f375-4d06-a205-0c0077bc12d9_isrForCrosstalkSources_995846_49.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a6e8572c-f375-4d06-a205-0c0077bc12d9_isrForCrosstalkSources_995846_49.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c395eabdd83c6c4e625a9b5b37826265 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/20037789-80e5-4f48-8c27-d976ab1c9905_isrForCrosstalkSources_995845_32.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/20037789-80e5-4f48-8c27-d976ab1c9905_isrForCrosstalkSources_995845_32.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b174fc225e56a6dfe00e5e347530610 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/f757448a-ac46-40e6-82b1-af738471ebdc_isrForCrosstalkSources_995990_29.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/f757448a-ac46-40e6-82b1-af738471ebdc_isrForCrosstalkSources_995990_29.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c8a9dbc1e5c5e1c7ddf75ed78cf959a with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ec9239ef-1374-416d-ae17-afba24b47fb8_isrForCrosstalkSources_995847_29.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ec9239ef-1374-416d-ae17-afba24b47fb8_isrForCrosstalkSources_995847_29.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f7569fbd5d06bd920603ca5b5d56d31 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b9ccc912-36c7-4f79-9227-dd75b487c75e_isrForCrosstalkSources_995992_57.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b9ccc912-36c7-4f79-9227-dd75b487c75e_isrForCrosstalkSources_995992_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:08 proc_lsst.multi:146 [INFO] found job 17063582 in provider astro proc_lsst.multi INFO: found job 17063582 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fae1d4b977ed0d1059c911c599906337 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/999b96ff-d1d4-4ca9-a610-0c0931b2e60d_isrForCrosstalkSources_995847_43.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/999b96ff-d1d4-4ca9-a610-0c0931b2e60d_isrForCrosstalkSources_995847_43.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd07ff6f5322937a8a92ac76ed30f36b with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47473b2d-5205-4c2c-a7c5-f1e262726222_isrForCrosstalkSources_995990_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47473b2d-5205-4c2c-a7c5-f1e262726222_isrForCrosstalkSources_995990_61.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebb557a157d01268b49b8666f84a9238 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/edeb2679-825f-491c-8432-d3b5dc72aa41_isrForCrosstalkSources_995846_50.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/edeb2679-825f-491c-8432-d3b5dc72aa41_isrForCrosstalkSources_995846_50.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36b0a5165bb1af4b3141bd15980633c0 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d41e0ca-7e7b-402a-ac87-d56340029283_isrForCrosstalkSources_995846_40.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5d41e0ca-7e7b-402a-ac87-d56340029283_isrForCrosstalkSources_995846_40.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b06ddf32936f74fad926b0d792bcf2a with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ba38b8e7-3da0-4e3e-857d-9fe5954339c9_isrForCrosstalkSources_995990_40.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ba38b8e7-3da0-4e3e-857d-9fe5954339c9_isrForCrosstalkSources_995990_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:10 proc_lsst.multi:146 [INFO] found job 17063583 in provider astro proc_lsst.multi INFO: found job 17063583 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1ac83acfe3342b80308e300a9731dd0 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5aaed2f5-1a0e-415b-843b-677e8da988b1_isrForCrosstalkSources_995845_10.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/5aaed2f5-1a0e-415b-843b-677e8da988b1_isrForCrosstalkSources_995845_10.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61d76ba775f1f08990fcfbe99936ace with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1b12c092-b9b7-453f-b8f5-764eba359194_isrForCrosstalkSources_995990_36.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/1b12c092-b9b7-453f-b8f5-764eba359194_isrForCrosstalkSources_995990_36.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcff8977b1be037450921396859f4938 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cfbc892a-59a9-4c34-9ee5-d054c73561d3_isrForCrosstalkSources_995845_4.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cfbc892a-59a9-4c34-9ee5-d054c73561d3_isrForCrosstalkSources_995845_4.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7514ea35187563a5f82ca68728d399ee with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b1ea726-d581-4a43-9b7e-e1595b25e998_isrForCrosstalkSources_995847_10.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b1ea726-d581-4a43-9b7e-e1595b25e998_isrForCrosstalkSources_995847_10.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b191335bb64b03f21e982096da3f665 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f9a38967-23be-41da-88bb-a785d6dfd49a_isrForCrosstalkSources_995992_13.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f9a38967-23be-41da-88bb-a785d6dfd49a_isrForCrosstalkSources_995992_13.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9481b62585091adf78069f00f6e119ea with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/48984ccc-1232-444c-8454-519f17dbf144_isrForCrosstalkSources_995846_59.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/48984ccc-1232-444c-8454-519f17dbf144_isrForCrosstalkSources_995846_59.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c192288382b438ab20d87a2cd15f2b32 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d0458f3d-bd46-472f-aea1-5e6d917a5a23_isrForCrosstalkSources_995846_16.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d0458f3d-bd46-472f-aea1-5e6d917a5a23_isrForCrosstalkSources_995846_16.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4279ebad791bead61e348997f61a49dc with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/550b5d96-6307-4c9b-99bb-6f83ec135ef5_isrForCrosstalkSources_995846_24.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/550b5d96-6307-4c9b-99bb-6f83ec135ef5_isrForCrosstalkSources_995846_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:12 proc_lsst.multi:146 [INFO] found job 17063584 in provider astro proc_lsst.multi INFO: found job 17063584 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d25433553041aa377933945d70f6da2 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d0bace7-cc17-42ee-808a-007ad421fc74_isrForCrosstalkSources_995992_22.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d0bace7-cc17-42ee-808a-007ad421fc74_isrForCrosstalkSources_995992_22.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316ed3fdf37fd133ee166c6c6a8e2a02 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bc4002b9-08ea-437d-a611-3c8ad16ad7db_isrForCrosstalkSources_995990_20.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bc4002b9-08ea-437d-a611-3c8ad16ad7db_isrForCrosstalkSources_995990_20.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93c33005a1e58d07184e1d4a4083326a with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d0151207-a303-41c9-81e3-1a634eec0d2c_isrForCrosstalkSources_995845_31.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d0151207-a303-41c9-81e3-1a634eec0d2c_isrForCrosstalkSources_995845_31.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e05a7e89634bd7e2e99df9fa8fd0bed with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4763d071-a06a-4bac-9fd5-4fdabeb1867b_isrForCrosstalkSources_995847_3.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4763d071-a06a-4bac-9fd5-4fdabeb1867b_isrForCrosstalkSources_995847_3.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8acf98311596e8140f4f75a710e2deb2 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/95319794-1484-4b48-ae8d-e7d5c5f14991_isrForCrosstalkSources_995990_51.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/95319794-1484-4b48-ae8d-e7d5c5f14991_isrForCrosstalkSources_995990_51.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9051f25bf04e16fb280b83d989dc4c0a with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/68418a59-00dc-4958-9172-bc7f1ddd213b_isrForCrosstalkSources_995991_34.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/68418a59-00dc-4958-9172-bc7f1ddd213b_isrForCrosstalkSources_995991_34.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 144e19a921ff9ee9a375b8fa617faee5 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1a70713b-fe95-4314-b038-842640d6a351_isrForCrosstalkSources_995991_39.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1a70713b-fe95-4314-b038-842640d6a351_isrForCrosstalkSources_995991_39.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 492b53c63e3dccefc1f84bb3c82ca67b with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fd7d70a3-5a6e-4c0e-b2e0-4f6caced8b55_isrForCrosstalkSources_995846_60.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fd7d70a3-5a6e-4c0e-b2e0-4f6caced8b55_isrForCrosstalkSources_995846_60.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db151ceab8b5c7e27b20b9423552bc2 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/08e7b07f-5959-4a66-8d95-e366154928e7_isrForCrosstalkSources_995991_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/08e7b07f-5959-4a66-8d95-e366154928e7_isrForCrosstalkSources_995991_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:14 proc_lsst.multi:146 [INFO] found job 17063585 in provider astro proc_lsst.multi INFO: found job 17063585 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:16 proc_lsst.multi:146 [INFO] found job 17063586 in provider astro proc_lsst.multi INFO: found job 17063586 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71e4e023b347e7529a278a824974f06b with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/939a2524-a544-43a2-a599-94c2f3df3ee4_isrForCrosstalkSources_995847_55.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/939a2524-a544-43a2-a599-94c2f3df3ee4_isrForCrosstalkSources_995847_55.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4abc77536c5726810c30af9ac1c1e44 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/abd4fb7b-db27-494b-bd25-341e563ff9ad_isrForCrosstalkSources_995846_38.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/abd4fb7b-db27-494b-bd25-341e563ff9ad_isrForCrosstalkSources_995846_38.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3720c35a6e220c3efd5ca5dd79cc658c with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b22b242b-fa90-42d9-b703-b714c5dd70ff_isrForCrosstalkSources_995847_32.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b22b242b-fa90-42d9-b703-b714c5dd70ff_isrForCrosstalkSources_995847_32.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1719b24402804a80695ba1f02bfa457 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/553a4c9b-b22d-40ad-9ef6-34ecb1c39ea3_isrForCrosstalkSources_995847_25.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/553a4c9b-b22d-40ad-9ef6-34ecb1c39ea3_isrForCrosstalkSources_995847_25.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0c8a8bd478aa91af35cdab7b8dabfe7 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bfd5d3c7-c191-4d23-b30d-6a339417d2d4_isrForCrosstalkSources_995992_4.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bfd5d3c7-c191-4d23-b30d-6a339417d2d4_isrForCrosstalkSources_995992_4.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3a46a9a80f309b9bb4711526f00b998 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b8027155-d4e2-44d7-9fc8-7364cd73853c_isrForCrosstalkSources_995992_28.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/b8027155-d4e2-44d7-9fc8-7364cd73853c_isrForCrosstalkSources_995992_28.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6855637603ef2b5bd88b7680b41d1670 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/74bf055e-09de-49e8-a6a7-38fbcc2cccf7_isrForCrosstalkSources_995990_37.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/74bf055e-09de-49e8-a6a7-38fbcc2cccf7_isrForCrosstalkSources_995990_37.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 634300fd7a82f287dcbc419a6b8a6bb8 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/cafa834c-21a1-4a32-8d25-72ff43a35a76_isrForCrosstalkSources_995846_3.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/cafa834c-21a1-4a32-8d25-72ff43a35a76_isrForCrosstalkSources_995846_3.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7dcd16b710bef49f572779aa9ccdb3a with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/9978b5c9-1776-400b-aa22-11ad065173f0_isrForCrosstalkSources_995846_43.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/9978b5c9-1776-400b-aa22-11ad065173f0_isrForCrosstalkSources_995846_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:18 proc_lsst.multi:146 [INFO] found job 17063587 in provider astro proc_lsst.multi INFO: found job 17063587 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecac84f2fb042b2537b91486b352fce8 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7ef89a59-7cee-4e30-8f6a-f644990202c9_isrForCrosstalkSources_995991_41.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7ef89a59-7cee-4e30-8f6a-f644990202c9_isrForCrosstalkSources_995991_41.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23e27d7f5547674cf142c34c45bbca54 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/70312bb8-3c0f-4299-a6b9-9e39ee9aa76d_isrForCrosstalkSources_995990_4.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/70312bb8-3c0f-4299-a6b9-9e39ee9aa76d_isrForCrosstalkSources_995990_4.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3407176945fa075d1cf065d3ae933d4d with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/20de8146-7df2-476f-8319-6673d9ee78d4_isrForCrosstalkSources_995990_6.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/20de8146-7df2-476f-8319-6673d9ee78d4_isrForCrosstalkSources_995990_6.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b89d6446ebbeafdc4032ba897093d4d with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/362509f6-f9a6-4548-b114-37a4b89ea9dd_isrForCrosstalkSources_995990_50.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/362509f6-f9a6-4548-b114-37a4b89ea9dd_isrForCrosstalkSources_995990_50.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 271d7ded37e7a0cecf0306d709e989b9 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4a543e29-da7b-4a19-ac20-c0f704b2337b_isrForCrosstalkSources_995992_19.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4a543e29-da7b-4a19-ac20-c0f704b2337b_isrForCrosstalkSources_995992_19.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4756cbc1c2e692f80386f23f1969a491 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c627918d-94ef-4b2a-9e7a-cd396cf442ba_isrForCrosstalkSources_995847_39.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/c627918d-94ef-4b2a-9e7a-cd396cf442ba_isrForCrosstalkSources_995847_39.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df511bc5fa059cff0e0949f1575761a5 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8787c5a6-761f-4830-9e77-b7d39e2f5299_isrForCrosstalkSources_995847_42.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/8787c5a6-761f-4830-9e77-b7d39e2f5299_isrForCrosstalkSources_995847_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:20 proc_lsst.multi:146 [INFO] found job 17063589 in provider astro proc_lsst.multi INFO: found job 17063589 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8495a51f8fbfd9fb49f0b27e27fe7ef5 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d4ff9c6c-bc13-4c15-9023-7dc223322b83_isrForCrosstalkSources_995845_53.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d4ff9c6c-bc13-4c15-9023-7dc223322b83_isrForCrosstalkSources_995845_53.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8466560a8adbeedd6f13be11334f375 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fe8f70e3-651f-4050-97db-0394d30381c9_isrForCrosstalkSources_995845_7.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/fe8f70e3-651f-4050-97db-0394d30381c9_isrForCrosstalkSources_995845_7.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c336afc84b6d9a183cf7333e47171b6f with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/4d29492c-e90e-4036-a3d1-ed0e39c402b8_isrForCrosstalkSources_995846_30.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/4d29492c-e90e-4036-a3d1-ed0e39c402b8_isrForCrosstalkSources_995846_30.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a3b77235ef4764a4b03aaa6fbf12fac with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/a96a6d28-c90f-4694-b2c0-0837e5ce27c0_isrForCrosstalkSources_995845_49.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/a96a6d28-c90f-4694-b2c0-0837e5ce27c0_isrForCrosstalkSources_995845_49.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b8fc9226cf392d28ea8bef4537b116a with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/46e591fb-badd-4bea-89cd-53591914ad30_isrForCrosstalkSources_995991_25.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/46e591fb-badd-4bea-89cd-53591914ad30_isrForCrosstalkSources_995991_25.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c42504efc028c746f2a43485263dce8c with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/bacca6fa-3258-4d38-a58d-5c5c78eafa92_isrForCrosstalkSources_995991_30.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/bacca6fa-3258-4d38-a58d-5c5c78eafa92_isrForCrosstalkSources_995991_30.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8b9ee27349fc45069f79e44f98ac4cf with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/311803fc-85ec-46a6-9ae8-2ed78a4b8752_isrForCrosstalkSources_995846_6.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/311803fc-85ec-46a6-9ae8-2ed78a4b8752_isrForCrosstalkSources_995846_6.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 857459503f4734f70ce579211415d9d9 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87a0c340-f872-41c0-a40d-977a90ddf653_isrForCrosstalkSources_995847_45.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/87a0c340-f872-41c0-a40d-977a90ddf653_isrForCrosstalkSources_995847_45.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeed284f0804c9cb3d592a50b7ebd7a8 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/14ebae4d-5611-4300-89cc-8ab243ecec76_isrForCrosstalkSources_995847_51.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/14ebae4d-5611-4300-89cc-8ab243ecec76_isrForCrosstalkSources_995847_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:23 proc_lsst.multi:146 [INFO] found job 17063590 in provider astro proc_lsst.multi INFO: found job 17063590 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b5f6d8371824ec9c3998be6662a3b02 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3689b240-8380-4035-b900-7b88eea872bf_isrForCrosstalkSources_995847_41.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3689b240-8380-4035-b900-7b88eea872bf_isrForCrosstalkSources_995847_41.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3be5d40bdf4ea621edcd0ef6c810262d with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ee129b28-1af9-4596-a968-b6d7aca60791_isrForCrosstalkSources_995992_37.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ee129b28-1af9-4596-a968-b6d7aca60791_isrForCrosstalkSources_995992_37.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93aa24ae4dbeceba84d002512a68d613 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/36eb4233-14c2-488f-bd9e-42ff8d6c4826_isrForCrosstalkSources_995847_8.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/36eb4233-14c2-488f-bd9e-42ff8d6c4826_isrForCrosstalkSources_995847_8.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d715fadaa24b656dff2901658e2585 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/15086afe-c0dd-4249-a4b2-e031ddf0d04d_isrForCrosstalkSources_995990_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/15086afe-c0dd-4249-a4b2-e031ddf0d04d_isrForCrosstalkSources_995990_13.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d36aff7b66eb1f2c788680b32037afc with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/797d3164-7c6b-4c8a-8f6c-84dd6a37e62f_isrForCrosstalkSources_995847_33.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/797d3164-7c6b-4c8a-8f6c-84dd6a37e62f_isrForCrosstalkSources_995847_33.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f125625e12713d48bd149226b65207b0 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ed77f56a-c121-482e-8ba1-5e6b57e96af0_isrForCrosstalkSources_995847_24.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ed77f56a-c121-482e-8ba1-5e6b57e96af0_isrForCrosstalkSources_995847_24.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2c1d0a5cab76a34621e9339b2b3bce9 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/37f5add3-0f05-4b8f-b825-ab77caa09031_isrForCrosstalkSources_995991_19.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/37f5add3-0f05-4b8f-b825-ab77caa09031_isrForCrosstalkSources_995991_19.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73a21d19fe50dcd0a7899862978e2cd with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6897091e-3c7e-47ee-afb5-385decc32d17_isrForCrosstalkSources_995846_55.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/6897091e-3c7e-47ee-afb5-385decc32d17_isrForCrosstalkSources_995846_55.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 185536ae5faa365c2d19bab7c9c42694 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/cae37bb9-9ae4-4f79-893c-6177f21008af_isrForCrosstalkSources_995990_56.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/cae37bb9-9ae4-4f79-893c-6177f21008af_isrForCrosstalkSources_995990_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:25 proc_lsst.multi:146 [INFO] found job 17063591 in provider astro proc_lsst.multi INFO: found job 17063591 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5650797d7c0908560444fe20c8b22c41 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/36b61023-ccc2-4326-ba0b-b49ca6e9bb70_isrForCrosstalkSources_995991_58.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/36b61023-ccc2-4326-ba0b-b49ca6e9bb70_isrForCrosstalkSources_995991_58.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a21fc23c6cbda3f70960323549b965f9 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/a91fd1e0-eee5-4868-8771-b00a8420e945_isrForCrosstalkSources_995991_57.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/a91fd1e0-eee5-4868-8771-b00a8420e945_isrForCrosstalkSources_995991_57.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9283f57d83fe1f306954b1816e2c9826 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7e458e5f-3481-4276-8733-7b9fa78fdecf_isrForCrosstalkSources_995990_57.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7e458e5f-3481-4276-8733-7b9fa78fdecf_isrForCrosstalkSources_995990_57.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fd99dff29db1a0e111a772ca457aba5 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e5bc199d-de3b-49a9-9a19-65a8cef80f62_isrForCrosstalkSources_995991_21.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/e5bc199d-de3b-49a9-9a19-65a8cef80f62_isrForCrosstalkSources_995991_21.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4bc73515e175abc98ed2d98155b4818 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/992a093d-7da8-43be-a705-d80f38335c9f_isrForCrosstalkSources_995990_1.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/992a093d-7da8-43be-a705-d80f38335c9f_isrForCrosstalkSources_995990_1.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdfc64fad7849566d0c08dc27164a350 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91430d59-bd32-46f1-a647-ce87f29c1533_isrForCrosstalkSources_995992_24.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/91430d59-bd32-46f1-a647-ce87f29c1533_isrForCrosstalkSources_995992_24.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7a9db950c39277053b64f76b3a88e53 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a1435403-3a2e-4c80-a5ea-2016185eb67c_isrForCrosstalkSources_995990_43.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/a1435403-3a2e-4c80-a5ea-2016185eb67c_isrForCrosstalkSources_995990_43.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efa352066924e56122684ff7a476d0d0 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/d0c73355-56ed-4adc-adcf-3c1a2eea55d3_isrForCrosstalkSources_995990_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/d0c73355-56ed-4adc-adcf-3c1a2eea55d3_isrForCrosstalkSources_995990_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:27 proc_lsst.multi:146 [INFO] found job 17063592 in provider astro proc_lsst.multi INFO: found job 17063592 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44e4fa0c7ecaa0815f672501d8f9f88d with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3bf16483-dc11-41ec-ae9a-7fa224b4bd5a_isrForCrosstalkSources_995846_31.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3bf16483-dc11-41ec-ae9a-7fa224b4bd5a_isrForCrosstalkSources_995846_31.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bffc608e54dbccb02cfff2cbace9845 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/4580211e-16b9-40a5-8f49-afaeb6ad7fbf_isrForCrosstalkSources_995845_55.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/4580211e-16b9-40a5-8f49-afaeb6ad7fbf_isrForCrosstalkSources_995845_55.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb927c93fc9b0499a3fd5e3c0a280eb with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/83976f11-4141-46ea-bbaf-5c55b867954f_isrForCrosstalkSources_995992_48.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/83976f11-4141-46ea-bbaf-5c55b867954f_isrForCrosstalkSources_995992_48.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4ccf488f1c7f069ff0d184e41b5d03a with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/12404c28-d096-4553-8c37-8ed600c7423b_isrForCrosstalkSources_995846_57.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/12404c28-d096-4553-8c37-8ed600c7423b_isrForCrosstalkSources_995846_57.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c83d8824c7586691277abe4ed474ba0c with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/5acd44af-f6c1-48f0-bcbf-2f9a83e01544_isrForCrosstalkSources_995991_16.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/5acd44af-f6c1-48f0-bcbf-2f9a83e01544_isrForCrosstalkSources_995991_16.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd014ddedf3391f00f4a395626485386 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/67fac9ff-3009-4bdf-a794-b5b540fc5ddd_isrForCrosstalkSources_995845_15.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/67fac9ff-3009-4bdf-a794-b5b540fc5ddd_isrForCrosstalkSources_995845_15.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f6662be393e3f18eb65e20244bc2a1 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07c728af-3105-4215-9c4e-fc2ccf934446_isrForCrosstalkSources_995845_20.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/07c728af-3105-4215-9c4e-fc2ccf934446_isrForCrosstalkSources_995845_20.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be382ee211aa921ed1a592c42ef5d75a with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/0f783341-ea76-496d-b22f-e6afd0a41970_isrForCrosstalkSources_995846_32.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/0f783341-ea76-496d-b22f-e6afd0a41970_isrForCrosstalkSources_995846_32.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c29ecca8c62924ffc57f2b8280af543 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3276e957-6a6f-4587-b2aa-d0eab4c2e89e_isrForCrosstalkSources_995846_46.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/3276e957-6a6f-4587-b2aa-d0eab4c2e89e_isrForCrosstalkSources_995846_46.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b86d35cb240ca31cc577d1b149aa7164 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47f20c36-a1ac-4878-9465-331cc452f900_isrForCrosstalkSources_995990_62.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/47f20c36-a1ac-4878-9465-331cc452f900_isrForCrosstalkSources_995990_62.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bcdd8f7a35a37f5ee56bba088ee94c2 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a3cd0ff-fe01-4843-a016-787312b112cf_isrForCrosstalkSources_995990_58.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a3cd0ff-fe01-4843-a016-787312b112cf_isrForCrosstalkSources_995990_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:29 proc_lsst.multi:146 [INFO] found job 17063593 in provider astro proc_lsst.multi INFO: found job 17063593 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 531e53d9521d066b5ef914c1de621905 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a83470d-cdc6-4181-807a-a18b1c5ef8eb_isrForCrosstalkSources_995990_46.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2a83470d-cdc6-4181-807a-a18b1c5ef8eb_isrForCrosstalkSources_995990_46.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2b6d82a9a21be85f198434b87fd0a5 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/49364ec9-0baa-4223-ae84-c1f7d9f7dbd5_isrForCrosstalkSources_995846_35.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/49364ec9-0baa-4223-ae84-c1f7d9f7dbd5_isrForCrosstalkSources_995846_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:31 proc_lsst.multi:146 [INFO] found job 17063594 in provider astro proc_lsst.multi INFO: found job 17063594 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 123437e6a1467895630d75832026ff3b with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a0bca795-b524-442f-9070-386b79964360_isrForCrosstalkSources_995846_20.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a0bca795-b524-442f-9070-386b79964360_isrForCrosstalkSources_995846_20.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00c1074b3228cff7fcbb19322a8a7b24 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/986a90b2-d4de-48a1-8df5-af854dc70fe2_isrForCrosstalkSources_995991_13.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/986a90b2-d4de-48a1-8df5-af854dc70fe2_isrForCrosstalkSources_995991_13.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ceaf236f3309da3b283cc5c0faf26b1 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1c3561dc-664b-4a7e-ab65-52804d8e0110_isrForCrosstalkSources_995991_8.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1c3561dc-664b-4a7e-ab65-52804d8e0110_isrForCrosstalkSources_995991_8.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3893542f851ae34e2edcb33ec2379b2 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/72137b87-d483-4dbd-bee3-9ae4421ccf0c_isrForCrosstalkSources_995992_18.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/72137b87-d483-4dbd-bee3-9ae4421ccf0c_isrForCrosstalkSources_995992_18.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d7af5a41b0561b9f57fd1c870b93f71 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/750c6ffd-8128-4f35-83ea-76f3245bf34f_isrForCrosstalkSources_995990_23.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/750c6ffd-8128-4f35-83ea-76f3245bf34f_isrForCrosstalkSources_995990_23.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb13326cee676e6535e3fb76c4daa737 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/53cb97ef-1194-4643-a713-4d9226e54ae9_isrForCrosstalkSources_995990_44.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/53cb97ef-1194-4643-a713-4d9226e54ae9_isrForCrosstalkSources_995990_44.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f5ff24c3cf57fc771be8c629f4fea85 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/044c679d-49a1-49b0-90e7-b241de305794_isrForCrosstalkSources_995990_52.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/044c679d-49a1-49b0-90e7-b241de305794_isrForCrosstalkSources_995990_52.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84037d47f47dd36f82f48225dd30278f with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbbda821-5574-4470-8e72-9fed2e130169_isrForCrosstalkSources_995990_45.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/bbbda821-5574-4470-8e72-9fed2e130169_isrForCrosstalkSources_995990_45.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5c2728505b4e65cd80c4dca6b5be594 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/85101d00-c5b5-45ab-88c6-701f6266cfb2_isrForCrosstalkSources_995847_23.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/85101d00-c5b5-45ab-88c6-701f6266cfb2_isrForCrosstalkSources_995847_23.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 468ce5dd55625e7f376bea12b012d1d6 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e9ecb920-fe13-44fa-ac6e-9b0f3a6e7d0e_isrForCrosstalkSources_995847_16.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e9ecb920-fe13-44fa-ac6e-9b0f3a6e7d0e_isrForCrosstalkSources_995847_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:33 proc_lsst.multi:146 [INFO] found job 17063595 in provider astro proc_lsst.multi INFO: found job 17063595 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7325ffed6533baa40460ceb158d0e41a with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/4166b755-091d-47d8-88cc-13a36d110659_isrForCrosstalkSources_995990_26.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/4166b755-091d-47d8-88cc-13a36d110659_isrForCrosstalkSources_995990_26.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65efa6a13c1f19045870482794ad88e9 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2c94c9aa-eb75-4318-908c-79758e1c2c84_isrForCrosstalkSources_995846_45.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2c94c9aa-eb75-4318-908c-79758e1c2c84_isrForCrosstalkSources_995846_45.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b18f73de97bad4df295649ccc010475e with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d7e2a07a-fc13-42fd-8481-2ef080408d1f_isrForCrosstalkSources_995845_62.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d7e2a07a-fc13-42fd-8481-2ef080408d1f_isrForCrosstalkSources_995845_62.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7ceaff7d45d418fe0e234b284690773 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2b565921-2d8b-4435-b5f5-46d4c6bf7bac_isrForCrosstalkSources_995846_51.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/2b565921-2d8b-4435-b5f5-46d4c6bf7bac_isrForCrosstalkSources_995846_51.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca36e8203ded787dc4aa725561139ac6 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/81afc037-3d06-41af-ab5b-33c7ed75d78a_isrForCrosstalkSources_995992_36.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/81afc037-3d06-41af-ab5b-33c7ed75d78a_isrForCrosstalkSources_995992_36.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1f26a8fcf4da136ea311837ceae38fb with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8b2d370-f5b3-4d03-b468-7892bccd9cef_isrForCrosstalkSources_995846_33.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8b2d370-f5b3-4d03-b468-7892bccd9cef_isrForCrosstalkSources_995846_33.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a8dd0641d9b0571e79f4515e565fe1b with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a23f5a-be22-4110-ae42-6fb728d1649e_isrForCrosstalkSources_995846_19.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a23f5a-be22-4110-ae42-6fb728d1649e_isrForCrosstalkSources_995846_19.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd9018b48a3544de73233e1af9c23556 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/18e0eb21-5ffd-4284-bd62-6d2d517e5106_isrForCrosstalkSources_995991_17.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/18e0eb21-5ffd-4284-bd62-6d2d517e5106_isrForCrosstalkSources_995991_17.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9184e76974ab368683bfbf8f9942bc9 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/de58121d-da04-4607-9869-bd3119840cc2_isrForCrosstalkSources_995990_28.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/de58121d-da04-4607-9869-bd3119840cc2_isrForCrosstalkSources_995990_28.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e039e7880652a4dd3b71c3c6a21a0813 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9f087ac1-bd99-4a5e-97dd-961df316b244_isrForCrosstalkSources_995991_40.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9f087ac1-bd99-4a5e-97dd-961df316b244_isrForCrosstalkSources_995991_40.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f519680f5501bbaeb304752db6cb9fa with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1f065fb2-09cb-4bab-9ae1-57f59399f7e4_isrForCrosstalkSources_995845_46.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/1f065fb2-09cb-4bab-9ae1-57f59399f7e4_isrForCrosstalkSources_995845_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:35 proc_lsst.multi:146 [INFO] found job 17063596 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063596 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4ab14ffe2c5912ec09adf0c2e61ecfa with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba59a3a1-eee0-41be-a50c-a1c062260f16_isrForCrosstalkSources_995847_2.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba59a3a1-eee0-41be-a50c-a1c062260f16_isrForCrosstalkSources_995847_2.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 990cdeed69e283cfa30730108f14c86f with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/50431d64-4e64-4414-9b19-0bb89dea3495_isrForCrosstalkSources_995992_58.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/50431d64-4e64-4414-9b19-0bb89dea3495_isrForCrosstalkSources_995992_58.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b77b45915810cef515dc5209c7bca2c with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a3ac5869-9bff-4b7f-a1c6-ab673140814d_isrForCrosstalkSources_995992_3.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a3ac5869-9bff-4b7f-a1c6-ab673140814d_isrForCrosstalkSources_995992_3.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3ddd3cd40cc4e5827d33ecc9b8fc6b7 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/63d52ae6-c85d-4801-8b2c-ab3aa691754d_isrForCrosstalkSources_995991_4.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/63d52ae6-c85d-4801-8b2c-ab3aa691754d_isrForCrosstalkSources_995991_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:38 proc_lsst.multi:146 [INFO] found job 17063597 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063597 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79fcb7cff87a1e39a959f0e8069b8cb5 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7216422-beda-49cc-9e94-714c58d246e8_isrForCrosstalkSources_995992_51.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/e7216422-beda-49cc-9e94-714c58d246e8_isrForCrosstalkSources_995992_51.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1d854804b9b1fe6d6414408abcccc2 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4e785b7a-7d80-4964-ad46-3749d62d625e_isrForCrosstalkSources_995991_51.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4e785b7a-7d80-4964-ad46-3749d62d625e_isrForCrosstalkSources_995991_51.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ddf2bdcd96582173aab06ef687c3ce4 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/203b2ef9-c229-44d3-b152-180ef1e466df_isrForCrosstalkSources_995990_16.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/203b2ef9-c229-44d3-b152-180ef1e466df_isrForCrosstalkSources_995990_16.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5092662e6556586b4a358a945c0d6e5e with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d1453187-fd81-4223-9e36-4a4b319da432_isrForCrosstalkSources_995845_28.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/d1453187-fd81-4223-9e36-4a4b319da432_isrForCrosstalkSources_995845_28.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b83b3fbbfb398bb1cb43fd815a6e832 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/faddacf8-8c5a-466c-8e80-878594863a0e_isrForCrosstalkSources_995991_26.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/faddacf8-8c5a-466c-8e80-878594863a0e_isrForCrosstalkSources_995991_26.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83e4788076573eaff865569b54fc54da with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/21d8ef43-9574-4697-8626-c5097d3ed8bf_isrForCrosstalkSources_995845_19.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/21d8ef43-9574-4697-8626-c5097d3ed8bf_isrForCrosstalkSources_995845_19.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b7a641f958f7ff5635aed34d5508f1d with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f7a9fcf7-cc00-4395-887c-7d7ad72740c3_isrForCrosstalkSources_995992_35.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/f7a9fcf7-cc00-4395-887c-7d7ad72740c3_isrForCrosstalkSources_995992_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:40 proc_lsst.multi:146 [INFO] found job 17063599 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063599 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09b2ac7b9d8e4441dd29a9ccd2063f6a with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/864d7158-a0b0-464d-828d-a7f0c0432ad4_isrForCrosstalkSources_995991_46.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/864d7158-a0b0-464d-828d-a7f0c0432ad4_isrForCrosstalkSources_995991_46.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca2a4dcb7a3cfa1f17fd2acccc4c5eb9 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1a7c36a-e331-40df-aab4-c2897829668a_isrForCrosstalkSources_995846_47.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/b1a7c36a-e331-40df-aab4-c2897829668a_isrForCrosstalkSources_995846_47.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc7de3e48ebe4a92e3e50f7575f030c9 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c39f8360-42c4-4e79-a355-d26e4d4181b9_isrForCrosstalkSources_995990_27.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c39f8360-42c4-4e79-a355-d26e4d4181b9_isrForCrosstalkSources_995990_27.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61e7ea882db102eceec1b31f37c7144e with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/afdb96a1-d173-4b46-805a-9439e4efd2f8_isrForCrosstalkSources_995847_58.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/afdb96a1-d173-4b46-805a-9439e4efd2f8_isrForCrosstalkSources_995847_58.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0e4f1435611a4a9f3dc2a45872060f with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7ca12956-745e-42a2-be58-c176051dc887_isrForCrosstalkSources_995992_14.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7ca12956-745e-42a2-be58-c176051dc887_isrForCrosstalkSources_995992_14.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64e6041717c306925621b2f47480ee94 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fb446150-0602-410d-93bb-d9948e612ca6_isrForCrosstalkSources_995992_55.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/fb446150-0602-410d-93bb-d9948e612ca6_isrForCrosstalkSources_995992_55.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf40134883be62c901d7149c51a2e03e with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/58de4f48-74ef-4592-8f88-ea1097cc190e_isrForCrosstalkSources_995991_52.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/58de4f48-74ef-4592-8f88-ea1097cc190e_isrForCrosstalkSources_995991_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:42 proc_lsst.multi:146 [INFO] found job 17063601 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063601 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b2031a874906b3a54f802fb37e4e531 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/817e6812-649b-40fb-830f-c452c52575d6_isrForCrosstalkSources_995992_15.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/817e6812-649b-40fb-830f-c452c52575d6_isrForCrosstalkSources_995992_15.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f571188760965fada687de5e8c90376a with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/96a140dd-2d74-4976-8d8a-dbf2304b5a20_isrForCrosstalkSources_995991_38.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/96a140dd-2d74-4976-8d8a-dbf2304b5a20_isrForCrosstalkSources_995991_38.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a6ea95469ceb104f261eac272d97f26 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/63019885-9cbb-4633-ba08-6946905d041e_isrForCrosstalkSources_995992_62.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/63019885-9cbb-4633-ba08-6946905d041e_isrForCrosstalkSources_995992_62.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2c7b7e08f179d4df953004900cf8515 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fc982387-6609-4642-8771-3c166b84e386_isrForCrosstalkSources_995990_5.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/fc982387-6609-4642-8771-3c166b84e386_isrForCrosstalkSources_995990_5.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fab091e125d36282f93579c83c236fc with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cda38f17-f7ae-41e0-8513-0a886f845ec0_isrForCrosstalkSources_995845_47.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cda38f17-f7ae-41e0-8513-0a886f845ec0_isrForCrosstalkSources_995845_47.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f69d7059603916c028b05cfec5743eb with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c25df085-3130-4e27-9d95-8f867182ec66_isrForCrosstalkSources_995990_53.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/c25df085-3130-4e27-9d95-8f867182ec66_isrForCrosstalkSources_995990_53.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 217d80b5c71d806014ee642242aeb429 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7c0f423c-d210-42a4-a917-a80e35dca88c_isrForCrosstalkSources_995990_12.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/7c0f423c-d210-42a4-a917-a80e35dca88c_isrForCrosstalkSources_995990_12.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f32b1c6594d9e7a2890ee81b1fba1110 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/02ff3113-9767-42ba-8ab8-4b74214c403a_isrForCrosstalkSources_995845_30.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/02ff3113-9767-42ba-8ab8-4b74214c403a_isrForCrosstalkSources_995845_30.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 446f7cf1479d7160595b9239a10a1b01 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fabc346b-d535-4915-ba5c-378ebf9cdd1d_isrForCrosstalkSources_995846_27.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/fabc346b-d535-4915-ba5c-378ebf9cdd1d_isrForCrosstalkSources_995846_27.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6535a0a8f9dec8bca8858f437c7394de with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ca366878-2efc-49b8-95cd-65850d491008_isrForCrosstalkSources_995990_47.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/ca366878-2efc-49b8-95cd-65850d491008_isrForCrosstalkSources_995990_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:44 proc_lsst.multi:146 [INFO] found job 17063602 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063602 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8cea64d7ff0f6d97ed86f03be235143 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3b5a15c2-a859-4940-a9ac-0106ba506c09_isrForCrosstalkSources_995992_50.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/3b5a15c2-a859-4940-a9ac-0106ba506c09_isrForCrosstalkSources_995992_50.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 420ca511061afb5277f082861ba4c512 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/0ffedcda-d487-454b-b258-3033dc338bec_isrForCrosstalkSources_995991_11.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/0ffedcda-d487-454b-b258-3033dc338bec_isrForCrosstalkSources_995991_11.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0400805d3ee3e92dd83cd81a231104c with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93213656-7d16-48a7-8211-3c69f74948ad_isrForCrosstalkSources_995992_41.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/93213656-7d16-48a7-8211-3c69f74948ad_isrForCrosstalkSources_995992_41.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 453264d37d9cb02fbd949ad41428b7e0 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/df364a44-805d-4e6e-9a0a-cdd322f4ca86_isrForCrosstalkSources_995991_47.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/df364a44-805d-4e6e-9a0a-cdd322f4ca86_isrForCrosstalkSources_995991_47.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691d386791d34aa1fa32bd1bfc635a01 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d2a175c1-4fa1-4955-8859-bccfd02db32e_isrForCrosstalkSources_995846_10.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d2a175c1-4fa1-4955-8859-bccfd02db32e_isrForCrosstalkSources_995846_10.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34163fb0b2b94fc671f146db3a708277 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b7abfa2-a664-42aa-a495-f2ad3cef60bd_isrForCrosstalkSources_995847_60.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/2b7abfa2-a664-42aa-a495-f2ad3cef60bd_isrForCrosstalkSources_995847_60.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd7f04e147a3003a8d3423b8191a8c2b with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/843bb7df-e126-449f-8802-22872cb44908_isrForCrosstalkSources_995847_5.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/843bb7df-e126-449f-8802-22872cb44908_isrForCrosstalkSources_995847_5.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3987b77ed959265259602adcbd4df5a with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d760c2d0-f25c-4f6f-89a3-518e14ab2272_isrForCrosstalkSources_995992_59.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/d760c2d0-f25c-4f6f-89a3-518e14ab2272_isrForCrosstalkSources_995992_59.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c808eb58a3b8a3a8400510986033e49 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ced5a832-59b9-48a1-b527-388dbc3228de_isrForCrosstalkSources_995847_48.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ced5a832-59b9-48a1-b527-388dbc3228de_isrForCrosstalkSources_995847_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:46 proc_lsst.multi:146 [INFO] found job 17063603 in provider escience_a40 proc_lsst.multi INFO: found job 17063603 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063603,17063605' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7179661a99e4f14d7c9a410c4aa65d81 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a1605ad8-bc4a-42f7-bebc-de8e4fa17e4e_isrForCrosstalkSources_995847_61.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a1605ad8-bc4a-42f7-bebc-de8e4fa17e4e_isrForCrosstalkSources_995847_61.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c70509dbf98b5a10fd64a8b53e966df9 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6cd3da14-50c0-4d1b-bbf6-54a8ae887d48_isrForCrosstalkSources_995992_46.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6cd3da14-50c0-4d1b-bbf6-54a8ae887d48_isrForCrosstalkSources_995992_46.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 074bc01d9b995358a23a8ca5324e3de5 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/48bcff86-f8d7-4718-a372-adf308de5569_isrForCrosstalkSources_995990_60.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/48bcff86-f8d7-4718-a372-adf308de5569_isrForCrosstalkSources_995990_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063605 R 17063603 R parsl.providers.slurm.slurm DEBUG: Updating job 17063605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063603 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:47 proc_lsst.multi:146 [INFO] found job 17063605 in provider escience_a40 proc_lsst.multi INFO: found job 17063605 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063603,17063605' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd50702db754937baa663b544b227240 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/0508290b-311f-41c3-818b-c81c923ee83e_isrForCrosstalkSources_995990_49.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/0508290b-311f-41c3-818b-c81c923ee83e_isrForCrosstalkSources_995990_49.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d8fa28fc0f32dff9718ae181961f88e with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3a355815-87eb-4a4b-8c22-3191789ee3f9_isrForCrosstalkSources_995990_15.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3a355815-87eb-4a4b-8c22-3191789ee3f9_isrForCrosstalkSources_995990_15.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71f388d12e62a3d52ad93ecbda7f9530 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7f8e638e-3d0e-48b4-9ef8-b6de66c5a09a_isrForCrosstalkSources_995991_45.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7f8e638e-3d0e-48b4-9ef8-b6de66c5a09a_isrForCrosstalkSources_995991_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063605 R 17063603 R parsl.providers.slurm.slurm DEBUG: Updating job 17063605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063603 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 81 parsl.dataflow.strategy DEBUG: Executor multi has 81 active tasks, 38/0 running/pending blocks, and 39 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22974949607440 2024-03-15 10:05:49 proc_lsst.multi:146 [INFO] found job 56858 in provider local proc_lsst.multi INFO: found job 56858 in provider local 2024-03-15 10:05:49 proc_lsst.multi:146 [INFO] found job 17063563 in provider astro proc_lsst.multi INFO: found job 17063563 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eccf731dcfa536e8252769581077936 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9a283d20-f1ba-484c-8098-9bfd8e50b3e9_isrForCrosstalkSources_995847_30.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9a283d20-f1ba-484c-8098-9bfd8e50b3e9_isrForCrosstalkSources_995847_30.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cc35c0829e02c296c7e6001033ae289 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/efe25367-f651-4904-8570-1d8a8f90d608_isrForCrosstalkSources_995845_37.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/efe25367-f651-4904-8570-1d8a8f90d608_isrForCrosstalkSources_995845_37.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1cc69472972fbed38d6ac915ad4c9c6 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/9ed339d3-27d7-4961-a253-212c9664cb45_isrForCrosstalkSources_995990_41.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/9ed339d3-27d7-4961-a253-212c9664cb45_isrForCrosstalkSources_995990_41.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8e247af83486f00c68037ac6e613f7e with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/9e3ef6f2-83b4-4d8e-bd52-4b8ed204b9d9_isrForCrosstalkSources_995992_60.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/9e3ef6f2-83b4-4d8e-bd52-4b8ed204b9d9_isrForCrosstalkSources_995992_60.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648ac96c4761937cb34c8ba3c4f87d92 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/135e2f36-bd6d-4fc4-8350-991c5f95e855_isrForCrosstalkSources_995990_21.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/135e2f36-bd6d-4fc4-8350-991c5f95e855_isrForCrosstalkSources_995990_21.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d1460edb9efc40ec49bae164932b711 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b7abea17-7a89-47ef-bc7e-fca11d3b76e8_isrForCrosstalkSources_995847_12.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b7abea17-7a89-47ef-bc7e-fca11d3b76e8_isrForCrosstalkSources_995847_12.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c9047143fb649aef7ba1dc2d1061da with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5da31c73-6d89-43b7-9806-dea62dbf38ba_isrForCrosstalkSources_995846_18.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/5da31c73-6d89-43b7-9806-dea62dbf38ba_isrForCrosstalkSources_995846_18.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f61308a277608a4e73704966ff50aebb with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/dc851891-5103-4228-92fe-b1e7b242750c_isrForCrosstalkSources_995845_42.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/dc851891-5103-4228-92fe-b1e7b242750c_isrForCrosstalkSources_995845_42.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6386b90b09a84d600c946a5d10fdd280 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/52496e44-5dbb-4f5e-b574-f1c9a8c3dba5_isrForCrosstalkSources_995845_23.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/52496e44-5dbb-4f5e-b574-f1c9a8c3dba5_isrForCrosstalkSources_995845_23.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aa7f87b5e32218af4eb9d9e353f2526 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1e327ac7-7368-49f7-aac8-b8a6c5b636c1_isrForCrosstalkSources_995991_9.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/1e327ac7-7368-49f7-aac8-b8a6c5b636c1_isrForCrosstalkSources_995991_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:51 proc_lsst.multi:146 [INFO] found job 17063564 in provider astro proc_lsst.multi INFO: found job 17063564 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5de46735d7f7cf7cec4dc692d783d680 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/eec3fab4-222a-4b5a-883f-b631ceb60bb8_isrForCrosstalkSources_995845_34.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/eec3fab4-222a-4b5a-883f-b631ceb60bb8_isrForCrosstalkSources_995845_34.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a47de161085e859e16f392d31da7cee with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df6892ec-435f-447c-9df3-8cedcedb127b_isrForCrosstalkSources_995846_21.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/df6892ec-435f-447c-9df3-8cedcedb127b_isrForCrosstalkSources_995846_21.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a85bbadf700515763e18dee37151682 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/045d9106-edae-4e92-a3c3-81baeede3a08_isrForCrosstalkSources_995992_44.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/045d9106-edae-4e92-a3c3-81baeede3a08_isrForCrosstalkSources_995992_44.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cc3da7255ff4cb49baf6735450e43a5 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a0cb0f-db5e-4636-aaff-03195b7e5cc3_isrForCrosstalkSources_995846_61.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/95a0cb0f-db5e-4636-aaff-03195b7e5cc3_isrForCrosstalkSources_995846_61.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e5fd33fed6d7410348a1499934a756 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/304b72ba-1249-4726-81a6-17e343932d65_isrForCrosstalkSources_995992_45.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/304b72ba-1249-4726-81a6-17e343932d65_isrForCrosstalkSources_995992_45.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2145ba78f90e7ab029a4d25abc179188 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c0da0851-b476-4ffc-b4ef-d1503e5d4eb3_isrForCrosstalkSources_995991_61.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/c0da0851-b476-4ffc-b4ef-d1503e5d4eb3_isrForCrosstalkSources_995991_61.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee5cd960cf61ed3eea7086257adcd0ca with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2c1fe54c-4fdd-4743-8b41-b633a018946e_isrForCrosstalkSources_995991_1.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/2c1fe54c-4fdd-4743-8b41-b633a018946e_isrForCrosstalkSources_995991_1.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 150453ceca7be4c199f00a1afd6a14c1 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/97c81f40-079c-4f5b-a649-148b0f92f3a7_isrForCrosstalkSources_995847_13.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/97c81f40-079c-4f5b-a649-148b0f92f3a7_isrForCrosstalkSources_995847_13.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39f0430099cde66324906c3dc25ac5e with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/92ebdacd-bc3b-4038-945f-08d9a53024bd_isrForCrosstalkSources_995847_31.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/92ebdacd-bc3b-4038-945f-08d9a53024bd_isrForCrosstalkSources_995847_31.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6984f25d9bc49f1a809a1286b13c8536 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2e2300ea-ba8f-4f70-83b6-7a5f2b9f733b_isrForCrosstalkSources_995992_61.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/2e2300ea-ba8f-4f70-83b6-7a5f2b9f733b_isrForCrosstalkSources_995992_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:53 proc_lsst.multi:146 [INFO] found job 17063565 in provider astro proc_lsst.multi INFO: found job 17063565 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1813f3afe470cbe0d8abb82bcea10621 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b733804a-8abf-438f-96b7-c575fc18e202_isrForCrosstalkSources_995847_50.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b733804a-8abf-438f-96b7-c575fc18e202_isrForCrosstalkSources_995847_50.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bf626bcfa16e461fbfa193dc4a8f243 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69e2303c-6c75-433b-87fc-07b9c6a19436_isrForCrosstalkSources_995846_53.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69e2303c-6c75-433b-87fc-07b9c6a19436_isrForCrosstalkSources_995846_53.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e27c807e0b301883f922927ca3ce635f with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/50a9214b-ea08-493a-a8ec-22dad3babcd1_isrForCrosstalkSources_995847_18.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/50a9214b-ea08-493a-a8ec-22dad3babcd1_isrForCrosstalkSources_995847_18.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32f4d86f3be53e7fc5035ec33cbbba3f with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/7443d55d-4203-4283-a897-637513683281_isrForCrosstalkSources_995845_40.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/7443d55d-4203-4283-a897-637513683281_isrForCrosstalkSources_995845_40.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d7e8e1f49fc5c3320095cb44f7e6163 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1d81c678-c569-47e3-8664-ed44706e37a5_isrForCrosstalkSources_995847_40.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/1d81c678-c569-47e3-8664-ed44706e37a5_isrForCrosstalkSources_995847_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:55 proc_lsst.multi:146 [INFO] found job 17063566 in provider astro proc_lsst.multi INFO: found job 17063566 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1616d7cfe9bf84b1ba661992e814be with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f0fa09ac-9315-4daa-a43b-988264e8e151_isrForCrosstalkSources_995845_57.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f0fa09ac-9315-4daa-a43b-988264e8e151_isrForCrosstalkSources_995845_57.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77e409c315b76ada6777af5ff9135b07 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/814d0c6c-51e4-4daf-a4a3-62086fb25133_isrForCrosstalkSources_995845_21.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/814d0c6c-51e4-4daf-a4a3-62086fb25133_isrForCrosstalkSources_995845_21.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9f77b21cbfd6a6ebf01b889301b7ef with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3dedd72b-3b16-4880-85a1-463074353a77_isrForCrosstalkSources_995990_42.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/3dedd72b-3b16-4880-85a1-463074353a77_isrForCrosstalkSources_995990_42.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae4a7b140385575ea9bf724cbef0a365 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b45d01b1-cad1-4d3f-9089-5474479b24d9_isrForCrosstalkSources_995991_14.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b45d01b1-cad1-4d3f-9089-5474479b24d9_isrForCrosstalkSources_995991_14.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e267e87be876aeab4e1fc79a6abd441 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bebcd5c3-5a50-4d5a-8b3d-5dc791e2fbf8_isrForCrosstalkSources_995992_7.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/bebcd5c3-5a50-4d5a-8b3d-5dc791e2fbf8_isrForCrosstalkSources_995992_7.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba7f35cc2a2d10ec7655ede32f0da39 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e0bcbb12-f44b-49d9-a508-131748fe47f8_isrForCrosstalkSources_995847_22.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/e0bcbb12-f44b-49d9-a508-131748fe47f8_isrForCrosstalkSources_995847_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:57 proc_lsst.multi:146 [INFO] found job 17063567 in provider astro proc_lsst.multi INFO: found job 17063567 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3bf82d222047a201b936e5d997f271 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9c992dc4-8552-4ab4-a05a-c394b0cef12c_isrForCrosstalkSources_995847_57.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/9c992dc4-8552-4ab4-a05a-c394b0cef12c_isrForCrosstalkSources_995847_57.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31888a535c76e00690f3f1806d3325dd with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9e06a40f-85fc-4c1a-9c23-e003380e2608_isrForCrosstalkSources_995991_48.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/9e06a40f-85fc-4c1a-9c23-e003380e2608_isrForCrosstalkSources_995991_48.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba2e3ad942a6b921e1c375a689913599 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/806ff80f-5ba6-4bea-a319-a3f006da5f0d_isrForCrosstalkSources_995992_30.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/806ff80f-5ba6-4bea-a319-a3f006da5f0d_isrForCrosstalkSources_995992_30.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc0521abafced793d7b948ad27d037fb with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4888c786-7f64-4ea7-8fc8-b19e91f491ca_isrForCrosstalkSources_995991_33.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/4888c786-7f64-4ea7-8fc8-b19e91f491ca_isrForCrosstalkSources_995991_33.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f125e2672d6fc060b3405676f8000077 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/54a442df-1e41-4376-a223-3c1b14d12394_isrForCrosstalkSources_995992_56.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/54a442df-1e41-4376-a223-3c1b14d12394_isrForCrosstalkSources_995992_56.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b141a27e994aae1d4d8cfa41c430c47b with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7acc35a0-40c7-4ef3-9842-346c8789975d_isrForCrosstalkSources_995992_12.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7acc35a0-40c7-4ef3-9842-346c8789975d_isrForCrosstalkSources_995992_12.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41c5a73104050df19b8aef44f0c7496b with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/26494850-a6fb-4665-ad39-8a957ce2fbdf_isrForCrosstalkSources_995845_41.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/26494850-a6fb-4665-ad39-8a957ce2fbdf_isrForCrosstalkSources_995845_41.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15fd483f4ce49ae9b139ed6d7b5a6164 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/77cef725-4ac9-412d-93f0-e6dae111a4a1_isrForCrosstalkSources_995991_62.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/77cef725-4ac9-412d-93f0-e6dae111a4a1_isrForCrosstalkSources_995991_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:05:59 proc_lsst.multi:146 [INFO] found job 17063568 in provider astro proc_lsst.multi INFO: found job 17063568 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6b36607740155a90bf65980810fd56c with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/bb88c9bd-294a-4b6a-9260-cda938e99388_isrForCrosstalkSources_995846_54.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/bb88c9bd-294a-4b6a-9260-cda938e99388_isrForCrosstalkSources_995846_54.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d127d8039b3bd38ac1684b18ebd1629 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2bb1ce69-6126-4791-bc0d-871b28417acd_isrForCrosstalkSources_995990_48.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/2bb1ce69-6126-4791-bc0d-871b28417acd_isrForCrosstalkSources_995990_48.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c126a0ebed61f21b8a07a96a94208324 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/1ca87e60-7b7b-4b69-b7fa-9b355cef242b_isrForCrosstalkSources_995846_34.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/1ca87e60-7b7b-4b69-b7fa-9b355cef242b_isrForCrosstalkSources_995846_34.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c08d4d18db9de7615eda6098aedd4f0 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3e52a3ee-7887-47d7-8d82-be0e9d47e8ce_isrForCrosstalkSources_995845_26.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/3e52a3ee-7887-47d7-8d82-be0e9d47e8ce_isrForCrosstalkSources_995845_26.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b979f1ee59b0711c63c13ee41ea4109 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/382f9849-e4db-470b-af5b-ee5e3da176b8_isrForCrosstalkSources_995847_37.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/382f9849-e4db-470b-af5b-ee5e3da176b8_isrForCrosstalkSources_995847_37.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f3dabecb36e4b9bf8dd1f4368898b4 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3023bbf5-1470-48d3-9aa7-fcd43740c881_isrForCrosstalkSources_995847_52.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/3023bbf5-1470-48d3-9aa7-fcd43740c881_isrForCrosstalkSources_995847_52.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e08b5ccee2ba62393984af153302bdfc with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b6bfa1f5-9b39-4ea7-8bd4-9f2f4bceb9e1_isrForCrosstalkSources_995847_62.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/b6bfa1f5-9b39-4ea7-8bd4-9f2f4bceb9e1_isrForCrosstalkSources_995847_62.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11d48b0afa07d12b2ff110616fa3fdf2 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f4c94185-263e-4d4b-8772-19a7312f64b4_isrForCrosstalkSources_995991_12.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/f4c94185-263e-4d4b-8772-19a7312f64b4_isrForCrosstalkSources_995991_12.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51285ab62a55307e2589de65c0e8b860 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/90d68a8e-4665-4f62-a46d-fe6cce5b3bee_isrForCrosstalkSources_995990_54.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995990/90d68a8e-4665-4f62-a46d-fe6cce5b3bee_isrForCrosstalkSources_995990_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:01 proc_lsst.multi:146 [INFO] found job 17063569 in provider astro proc_lsst.multi INFO: found job 17063569 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b6ac0c0925a281f5d7cb675c0c92910 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba914760-b173-4589-922e-1ac95e3ba2dc_isrForCrosstalkSources_995847_6.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ba914760-b173-4589-922e-1ac95e3ba2dc_isrForCrosstalkSources_995847_6.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3034fef8b17041ca5885bdc8bacd0bca with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f21b252a-0b80-4e60-87ce-09f9f9bfac22_isrForCrosstalkSources_995845_5.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/f21b252a-0b80-4e60-87ce-09f9f9bfac22_isrForCrosstalkSources_995845_5.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1b51cba0696290da363ed342ac95b34 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/61f8c355-60a2-4d96-82e3-864664fcea9b_isrForCrosstalkSources_995992_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/61f8c355-60a2-4d96-82e3-864664fcea9b_isrForCrosstalkSources_995992_25.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f58cd93403930d6bf5e6bb9efe8499 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d7043dc-a790-4ab5-81cb-8ab666cfd9b0_isrForCrosstalkSources_995992_11.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/4d7043dc-a790-4ab5-81cb-8ab666cfd9b0_isrForCrosstalkSources_995992_11.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd10635ea611306358f8bf1fb8863710 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a16813b4-c791-4591-b879-64a38ffd2ac4_isrForCrosstalkSources_995847_36.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/a16813b4-c791-4591-b879-64a38ffd2ac4_isrForCrosstalkSources_995847_36.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f856cb977c02da89724ee4f1b83a7aac with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a9b99d6a-980d-4a82-a6dc-f39435fe7b14_isrForCrosstalkSources_995992_52.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/a9b99d6a-980d-4a82-a6dc-f39435fe7b14_isrForCrosstalkSources_995992_52.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9d90e76a9fea9d1e0809652e9c8554 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a740a5cb-c0d9-46cd-b93e-407efacad0da_isrForCrosstalkSources_995846_37.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a740a5cb-c0d9-46cd-b93e-407efacad0da_isrForCrosstalkSources_995846_37.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8ddd606fcc66e1fc6ffe7e6f7d7a897 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ccebc81f-c58c-4fc1-95b0-c11b2e7bc945_isrForCrosstalkSources_995992_21.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/ccebc81f-c58c-4fc1-95b0-c11b2e7bc945_isrForCrosstalkSources_995992_21.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 629c5ace16e8267b668c13be57a6196b with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cde90d42-aa7c-472e-95f5-beadcb253dee_isrForCrosstalkSources_995845_38.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cde90d42-aa7c-472e-95f5-beadcb253dee_isrForCrosstalkSources_995845_38.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec69f97d36cb69c02ec81eeaec09b124 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/94202a2c-1b56-4f9f-8400-6290a4ef8a61_isrForCrosstalkSources_995847_26.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/94202a2c-1b56-4f9f-8400-6290a4ef8a61_isrForCrosstalkSources_995847_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:03 proc_lsst.multi:146 [INFO] found job 17063571 in provider astro proc_lsst.multi INFO: found job 17063571 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38809c140049a9e0844765a77b97fd2b with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8a0d037-b9cc-404e-8514-cd9c8d8a0cf7_isrForCrosstalkSources_995846_56.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/d8a0d037-b9cc-404e-8514-cd9c8d8a0cf7_isrForCrosstalkSources_995846_56.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef04189dcaad6243a8cd84135a6793ea with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4375044b-b4b0-4592-9065-d2b6901b6af8_isrForCrosstalkSources_995847_19.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/4375044b-b4b0-4592-9065-d2b6901b6af8_isrForCrosstalkSources_995847_19.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 911635597f21054145a291b541180bee with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/7f8ff80b-9bd4-4db2-9241-a0c6517e71fc_isrForCrosstalkSources_995846_12.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/7f8ff80b-9bd4-4db2-9241-a0c6517e71fc_isrForCrosstalkSources_995846_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:05 proc_lsst.multi:146 [INFO] found job 17063572 in provider astro proc_lsst.multi INFO: found job 17063572 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18214c07497f6b4be5666a034f39095b with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b4255a5e-277c-4b96-adf7-03fcbcfa32c0_isrForCrosstalkSources_995991_29.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/b4255a5e-277c-4b96-adf7-03fcbcfa32c0_isrForCrosstalkSources_995991_29.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 332a1551b8e581f321287bdb5805100f with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/50cc8441-b41c-4fc2-8777-f07f61a01183_isrForCrosstalkSources_995991_37.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/50cc8441-b41c-4fc2-8777-f07f61a01183_isrForCrosstalkSources_995991_37.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee3af5415f135e2f84ecde53045b3594 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/22ea96b8-cba3-4c81-93a7-2c2971387b9e_isrForCrosstalkSources_995992_31.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/22ea96b8-cba3-4c81-93a7-2c2971387b9e_isrForCrosstalkSources_995992_31.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ff78cf6ca73c79a083e493beb991b64 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/924fb05d-4469-48f6-9177-aec025fce00e_isrForCrosstalkSources_995847_34.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/924fb05d-4469-48f6-9177-aec025fce00e_isrForCrosstalkSources_995847_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:07 proc_lsst.multi:146 [INFO] found job 17063573 in provider astro proc_lsst.multi INFO: found job 17063573 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acce224739edbe0db6ef18bf77fc6d56 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/531b193b-e1b6-4189-a151-f0496d0a5b67_isrForCrosstalkSources_995845_1.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/531b193b-e1b6-4189-a151-f0496d0a5b67_isrForCrosstalkSources_995845_1.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb330cbf44d771466988991ac7b3faac with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7aa3aa16-1f97-436e-b5f1-afe82a834446_isrForCrosstalkSources_995992_38.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/7aa3aa16-1f97-436e-b5f1-afe82a834446_isrForCrosstalkSources_995992_38.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ab80ce855ce9e930e400a4d179d34b6 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/626fe7a2-3677-4b6d-8836-20cca4392c44_isrForCrosstalkSources_995847_11.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/626fe7a2-3677-4b6d-8836-20cca4392c44_isrForCrosstalkSources_995847_11.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d279f4dc6bb909c1400dbc6d83c5a41 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/befd785b-c2b3-4d47-9bb0-2d2f4a694565_isrForCrosstalkSources_995992_43.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/befd785b-c2b3-4d47-9bb0-2d2f4a694565_isrForCrosstalkSources_995992_43.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e3018bacd4afd8ea767423de029e537 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/28f298ea-dd3e-4e41-a321-746fd2cbfdcc_isrForCrosstalkSources_995845_51.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/28f298ea-dd3e-4e41-a321-746fd2cbfdcc_isrForCrosstalkSources_995845_51.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 754d254b1a6734db75e19701c3e29f27 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/228d6d32-2cdf-4ece-b959-aef1b76f417b_isrForCrosstalkSources_995991_54.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/228d6d32-2cdf-4ece-b959-aef1b76f417b_isrForCrosstalkSources_995991_54.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04c9b5a892b4670542d7407be452d1cd with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6002d2dc-79a3-4775-abf4-be7c4ff4d423_isrForCrosstalkSources_995992_5.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/6002d2dc-79a3-4775-abf4-be7c4ff4d423_isrForCrosstalkSources_995992_5.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30c5d279b43f1d136621fb9fb4df872b with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a1e45d64-5b78-4cfc-b956-a6027c31e7e7_isrForCrosstalkSources_995846_8.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/a1e45d64-5b78-4cfc-b956-a6027c31e7e7_isrForCrosstalkSources_995846_8.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bd84f03ab62419ab5059c79885366bc with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ff1dc257-c9f8-4a96-b349-a654d20dfad7_isrForCrosstalkSources_995847_9.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/ff1dc257-c9f8-4a96-b349-a654d20dfad7_isrForCrosstalkSources_995847_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:08 proc_lsst.multi:146 [INFO] found job 17063574 in provider astro proc_lsst.multi INFO: found job 17063574 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:08 proc_lsst.multi:146 [INFO] found job 17063575 in provider astro proc_lsst.multi INFO: found job 17063575 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 338a7ede98d0acd4594aa903316f8e15 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ac719d69-5556-45f8-a87a-ec612c2057fc_isrForCrosstalkSources_995845_59.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/ac719d69-5556-45f8-a87a-ec612c2057fc_isrForCrosstalkSources_995845_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:09 proc_lsst.multi:146 [INFO] found job 17063577 in provider astro proc_lsst.multi INFO: found job 17063577 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:09 proc_lsst.multi:146 [INFO] found job 17063578 in provider astro proc_lsst.multi INFO: found job 17063578 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:09 proc_lsst.multi:146 [INFO] found job 17063579 in provider astro proc_lsst.multi INFO: found job 17063579 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd18825c3d19647b7eb60f80f7699f1 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/33a03a2a-6ef4-425d-aef2-ab8fbc1a1cf9_isrForCrosstalkSources_995845_13.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/33a03a2a-6ef4-425d-aef2-ab8fbc1a1cf9_isrForCrosstalkSources_995845_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:09 proc_lsst.multi:146 [INFO] found job 17063580 in provider astro proc_lsst.multi INFO: found job 17063580 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:09 proc_lsst.multi:146 [INFO] found job 17063581 in provider astro proc_lsst.multi INFO: found job 17063581 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:10 proc_lsst.multi:146 [INFO] found job 17063582 in provider astro proc_lsst.multi INFO: found job 17063582 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba94d6b869b84189a3bd056c63dc32df with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/6d0789ef-a2bd-4879-ac80-e3c2773f5c05_isrForCrosstalkSources_995847_7.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995847/6d0789ef-a2bd-4879-ac80-e3c2773f5c05_isrForCrosstalkSources_995847_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:10 proc_lsst.multi:146 [INFO] found job 17063583 in provider astro proc_lsst.multi INFO: found job 17063583 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:10 proc_lsst.multi:146 [INFO] found job 17063584 in provider astro proc_lsst.multi INFO: found job 17063584 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428c8660632bba91cdfb76d85457cfce with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7d32c36a-8567-464e-afa7-2691b130e52a_isrForCrosstalkSources_995991_43.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995991/7d32c36a-8567-464e-afa7-2691b130e52a_isrForCrosstalkSources_995991_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:10 proc_lsst.multi:146 [INFO] found job 17063585 in provider astro proc_lsst.multi INFO: found job 17063585 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:11 proc_lsst.multi:146 [INFO] found job 17063586 in provider astro proc_lsst.multi INFO: found job 17063586 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:11 proc_lsst.multi:146 [INFO] found job 17063587 in provider astro proc_lsst.multi INFO: found job 17063587 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d74f1d2f8074f9e9d8abf725190fc19b with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cc0949d0-b847-4883-ad57-c213d5acfb4f_isrForCrosstalkSources_995845_50.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995845/cc0949d0-b847-4883-ad57-c213d5acfb4f_isrForCrosstalkSources_995845_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:11 proc_lsst.multi:146 [INFO] found job 17063589 in provider astro proc_lsst.multi INFO: found job 17063589 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:11 proc_lsst.multi:146 [INFO] found job 17063590 in provider astro proc_lsst.multi INFO: found job 17063590 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f21abdde8c80b8c36064a2ab7b9d78f1 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69d81174-993b-4546-aa02-794ca3e90d8e_isrForCrosstalkSources_995846_17.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995846/69d81174-993b-4546-aa02-794ca3e90d8e_isrForCrosstalkSources_995846_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:11 proc_lsst.multi:146 [INFO] found job 17063591 in provider astro proc_lsst.multi INFO: found job 17063591 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:12 proc_lsst.multi:146 [INFO] found job 17063592 in provider astro proc_lsst.multi INFO: found job 17063592 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:12 proc_lsst.multi:146 [INFO] found job 17063593 in provider astro proc_lsst.multi INFO: found job 17063593 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:12 proc_lsst.multi:146 [INFO] found job 17063594 in provider astro proc_lsst.multi INFO: found job 17063594 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:12 proc_lsst.multi:146 [INFO] found job 17063595 in provider astro proc_lsst.multi INFO: found job 17063595 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063563,17063564,17063565,17063566,17063567,17063568,17063569,17063571,17063572,17063573,17063574,17063575,17063577,17063578,17063579,17063580,17063581,17063582,17063583,17063584,17063585,17063586,17063587,17063589,17063590,17063591,17063592,17063593,17063594,17063595' parsl.providers.slurm.slurm DEBUG: squeue returned 17063595 R 17063594 R 17063593 R 17063591 R 17063592 R 17063589 R 17063590 R 17063585 R 17063586 R 17063587 R 17063583 R 17063584 R 17063578 R 17063579 R 17063580 R 17063581 R 17063582 R 17063575 R 17063577 R 17063572 R 17063573 R 17063574 R 17063571 R 17063564 R 17063565 R 17063566 R 17063567 R 17063568 R 17063569 R 17063563 R parsl.providers.slurm.slurm DEBUG: Updating job 17063595 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063594 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063593 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063591 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063592 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063589 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063590 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063585 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063586 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063587 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063583 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063584 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063578 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063579 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063580 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063581 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063582 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063575 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063577 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063572 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063573 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063574 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063571 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063564 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063565 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063566 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063567 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063568 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063569 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063563 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:13 proc_lsst.multi:146 [INFO] found job 17063596 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063596 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:13 proc_lsst.multi:146 [INFO] found job 17063597 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063597 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af159f0dc5c2a2aee6c42b5eb954f342 with result from task 335 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/51bb0312-5c23-4f33-a299-a9da16f1b5a1_isrForCrosstalkSources_995992_47.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/submit/DEEP/20210510/A1c/science#step0/20240315T100328Z/logs/isrForCrosstalkSources/995992/51bb0312-5c23-4f33-a299-a9da16f1b5a1_isrForCrosstalkSources_995992_47.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:13 proc_lsst.multi:146 [INFO] found job 17063599 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063599 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:13 proc_lsst.multi:146 [INFO] found job 17063601 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063601 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:14 proc_lsst.multi:146 [INFO] found job 17063602 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063602 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063596,17063597,17063599,17063601,17063602' parsl.providers.slurm.slurm DEBUG: squeue returned 17063602 R 17063601 R 17063599 R 17063597 R 17063596 R parsl.providers.slurm.slurm DEBUG: Updating job 17063602 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063601 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063599 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063597 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063596 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:14 proc_lsst.multi:146 [INFO] found job 17063603 in provider escience_a40 proc_lsst.multi INFO: found job 17063603 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063603,17063605' parsl.providers.slurm.slurm DEBUG: squeue returned 17063605 R 17063603 R parsl.providers.slurm.slurm DEBUG: Updating job 17063605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063603 with slurm status R to parsl state JobState.RUNNING 2024-03-15 10:06:14 proc_lsst.multi:146 [INFO] found job 17063605 in provider escience_a40 proc_lsst.multi INFO: found job 17063605 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17063603,17063605' parsl.providers.slurm.slurm DEBUG: squeue returned 17063605 R 17063603 R parsl.providers.slurm.slurm DEBUG: Updating job 17063605 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17063603 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 38/0 running/pending blocks, and 39 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (38) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 2.384185791015625e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22974949607440 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=38, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 38 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62e78b2298b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62e78b2298b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9746d0c04c61 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9746d0c04c61 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5901615bab2d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5901615bab2d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5df89b77ad8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5df89b77ad8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a466e3b55988 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a466e3b55988 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5fb58c8d14bb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5fb58c8d14bb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b8d465b8392f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b8d465b8392f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e02abe544217 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e02abe544217 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 330a8fd7ec7d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 330a8fd7ec7d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 748710b467d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 748710b467d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aad12c17ac5e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aad12c17ac5e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0e6dd1342adc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0e6dd1342adc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e5c82550774 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e5c82550774 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7b473447ae5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7b473447ae5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0ea0037b53a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0ea0037b53a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 66104ebccd76 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 66104ebccd76 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffa49d3b3eeb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffa49d3b3eeb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fcbefebe650a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fcbefebe650a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c041e50ef89 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c041e50ef89 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1930bd551f67 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1930bd551f67 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0b06f926c02c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0b06f926c02c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 723093610831 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 723093610831 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8bb40d2c2b28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8bb40d2c2b28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8b60b654d8ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8b60b654d8ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51a1a7f01312 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51a1a7f01312 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: edc08ef94ee9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: edc08ef94ee9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2c00d204b17 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2c00d204b17 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e71022900272 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e71022900272 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2e4f2494d8d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2e4f2494d8d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6df51c2bdbe4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6df51c2bdbe4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a501e4666fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a501e4666fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94197b3865d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94197b3865d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5c86b6bee798 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5c86b6bee798 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19c79491b7bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19c79491b7bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e023a10ca5e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e023a10ca5e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e5048110195a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e5048110195a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 961e7947b34f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 961e7947b34f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd0d26efed68 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd0d26efed68 2024-03-15 10:06:14 proc_lsst.multi:146 [INFO] found job 17063601 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063601 in provider escience_rtx6k 2024-03-15 10:06:14 proc_lsst.multi:201 [INFO] cancelling 17063601 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063601 on provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:146 [INFO] found job 17063596 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063596 in provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:201 [INFO] cancelling 17063596 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063596 on provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:146 [INFO] found job 17063599 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063599 in provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:201 [INFO] cancelling 17063599 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063599 on provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:146 [INFO] found job 17063597 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063597 in provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:201 [INFO] cancelling 17063597 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063597 on provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:146 [INFO] found job 17063602 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063602 in provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:201 [INFO] cancelling 17063602 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063602 on provider escience_rtx6k 2024-03-15 10:06:15 proc_lsst.multi:146 [INFO] found job 56858 in provider local proc_lsst.multi INFO: found job 56858 in provider local 2024-03-15 10:06:15 proc_lsst.multi:201 [INFO] cancelling 56858 on provider local proc_lsst.multi INFO: cancelling 56858 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 56858 2024-03-15 10:06:15 proc_lsst.multi:146 [INFO] found job 17063574 in provider astro proc_lsst.multi INFO: found job 17063574 in provider astro 2024-03-15 10:06:15 proc_lsst.multi:201 [INFO] cancelling 17063574 on provider astro proc_lsst.multi INFO: cancelling 17063574 on provider astro 2024-03-15 10:06:16 proc_lsst.multi:146 [INFO] found job 17063603 in provider escience_a40 proc_lsst.multi INFO: found job 17063603 in provider escience_a40 2024-03-15 10:06:16 proc_lsst.multi:201 [INFO] cancelling 17063603 on provider escience_a40 proc_lsst.multi INFO: cancelling 17063603 on provider escience_a40 2024-03-15 10:06:16 proc_lsst.multi:146 [INFO] found job 17063573 in provider astro proc_lsst.multi INFO: found job 17063573 in provider astro 2024-03-15 10:06:16 proc_lsst.multi:201 [INFO] cancelling 17063573 on provider astro proc_lsst.multi INFO: cancelling 17063573 on provider astro 2024-03-15 10:06:16 proc_lsst.multi:146 [INFO] found job 17063595 in provider astro proc_lsst.multi INFO: found job 17063595 in provider astro 2024-03-15 10:06:16 proc_lsst.multi:201 [INFO] cancelling 17063595 on provider astro proc_lsst.multi INFO: cancelling 17063595 on provider astro 2024-03-15 10:06:16 proc_lsst.multi:146 [INFO] found job 17063572 in provider astro proc_lsst.multi INFO: found job 17063572 in provider astro 2024-03-15 10:06:16 proc_lsst.multi:201 [INFO] cancelling 17063572 on provider astro proc_lsst.multi INFO: cancelling 17063572 on provider astro 2024-03-15 10:06:16 proc_lsst.multi:146 [INFO] found job 17063575 in provider astro proc_lsst.multi INFO: found job 17063575 in provider astro 2024-03-15 10:06:16 proc_lsst.multi:201 [INFO] cancelling 17063575 on provider astro proc_lsst.multi INFO: cancelling 17063575 on provider astro 2024-03-15 10:06:17 proc_lsst.multi:146 [INFO] found job 17063577 in provider astro proc_lsst.multi INFO: found job 17063577 in provider astro 2024-03-15 10:06:17 proc_lsst.multi:201 [INFO] cancelling 17063577 on provider astro proc_lsst.multi INFO: cancelling 17063577 on provider astro 2024-03-15 10:06:17 proc_lsst.multi:146 [INFO] found job 17063590 in provider astro proc_lsst.multi INFO: found job 17063590 in provider astro 2024-03-15 10:06:17 proc_lsst.multi:201 [INFO] cancelling 17063590 on provider astro proc_lsst.multi INFO: cancelling 17063590 on provider astro 2024-03-15 10:06:17 proc_lsst.multi:146 [INFO] found job 17063589 in provider astro proc_lsst.multi INFO: found job 17063589 in provider astro 2024-03-15 10:06:17 proc_lsst.multi:201 [INFO] cancelling 17063589 on provider astro proc_lsst.multi INFO: cancelling 17063589 on provider astro 2024-03-15 10:06:17 proc_lsst.multi:146 [INFO] found job 17063605 in provider escience_a40 proc_lsst.multi INFO: found job 17063605 in provider escience_a40 2024-03-15 10:06:17 proc_lsst.multi:201 [INFO] cancelling 17063605 on provider escience_a40 proc_lsst.multi INFO: cancelling 17063605 on provider escience_a40 2024-03-15 10:06:17 proc_lsst.multi:146 [INFO] found job 17063583 in provider astro proc_lsst.multi INFO: found job 17063583 in provider astro 2024-03-15 10:06:17 proc_lsst.multi:201 [INFO] cancelling 17063583 on provider astro proc_lsst.multi INFO: cancelling 17063583 on provider astro 2024-03-15 10:06:18 proc_lsst.multi:146 [INFO] found job 17063593 in provider astro proc_lsst.multi INFO: found job 17063593 in provider astro 2024-03-15 10:06:18 proc_lsst.multi:201 [INFO] cancelling 17063593 on provider astro proc_lsst.multi INFO: cancelling 17063593 on provider astro 2024-03-15 10:06:18 proc_lsst.multi:146 [INFO] found job 17063584 in provider astro proc_lsst.multi INFO: found job 17063584 in provider astro 2024-03-15 10:06:18 proc_lsst.multi:201 [INFO] cancelling 17063584 on provider astro proc_lsst.multi INFO: cancelling 17063584 on provider astro 2024-03-15 10:06:18 proc_lsst.multi:146 [INFO] found job 17063571 in provider astro proc_lsst.multi INFO: found job 17063571 in provider astro 2024-03-15 10:06:18 proc_lsst.multi:201 [INFO] cancelling 17063571 on provider astro proc_lsst.multi INFO: cancelling 17063571 on provider astro 2024-03-15 10:06:18 proc_lsst.multi:146 [INFO] found job 17063586 in provider astro proc_lsst.multi INFO: found job 17063586 in provider astro 2024-03-15 10:06:18 proc_lsst.multi:201 [INFO] cancelling 17063586 on provider astro proc_lsst.multi INFO: cancelling 17063586 on provider astro 2024-03-15 10:06:18 proc_lsst.multi:146 [INFO] found job 17063568 in provider astro proc_lsst.multi INFO: found job 17063568 in provider astro 2024-03-15 10:06:18 proc_lsst.multi:201 [INFO] cancelling 17063568 on provider astro proc_lsst.multi INFO: cancelling 17063568 on provider astro 2024-03-15 10:06:19 proc_lsst.multi:146 [INFO] found job 17063585 in provider astro proc_lsst.multi INFO: found job 17063585 in provider astro 2024-03-15 10:06:19 proc_lsst.multi:201 [INFO] cancelling 17063585 on provider astro proc_lsst.multi INFO: cancelling 17063585 on provider astro 2024-03-15 10:06:19 proc_lsst.multi:146 [INFO] found job 17063569 in provider astro proc_lsst.multi INFO: found job 17063569 in provider astro 2024-03-15 10:06:19 proc_lsst.multi:201 [INFO] cancelling 17063569 on provider astro proc_lsst.multi INFO: cancelling 17063569 on provider astro 2024-03-15 10:06:19 proc_lsst.multi:146 [INFO] found job 17063563 in provider astro proc_lsst.multi INFO: found job 17063563 in provider astro 2024-03-15 10:06:19 proc_lsst.multi:201 [INFO] cancelling 17063563 on provider astro proc_lsst.multi INFO: cancelling 17063563 on provider astro 2024-03-15 10:06:19 proc_lsst.multi:146 [INFO] found job 17063565 in provider astro proc_lsst.multi INFO: found job 17063565 in provider astro 2024-03-15 10:06:19 proc_lsst.multi:201 [INFO] cancelling 17063565 on provider astro proc_lsst.multi INFO: cancelling 17063565 on provider astro 2024-03-15 10:06:20 proc_lsst.multi:146 [INFO] found job 17063587 in provider astro proc_lsst.multi INFO: found job 17063587 in provider astro 2024-03-15 10:06:20 proc_lsst.multi:201 [INFO] cancelling 17063587 on provider astro proc_lsst.multi INFO: cancelling 17063587 on provider astro 2024-03-15 10:06:20 proc_lsst.multi:146 [INFO] found job 17063566 in provider astro proc_lsst.multi INFO: found job 17063566 in provider astro 2024-03-15 10:06:20 proc_lsst.multi:201 [INFO] cancelling 17063566 on provider astro proc_lsst.multi INFO: cancelling 17063566 on provider astro 2024-03-15 10:06:20 proc_lsst.multi:146 [INFO] found job 17063567 in provider astro proc_lsst.multi INFO: found job 17063567 in provider astro 2024-03-15 10:06:20 proc_lsst.multi:201 [INFO] cancelling 17063567 on provider astro proc_lsst.multi INFO: cancelling 17063567 on provider astro 2024-03-15 10:06:20 proc_lsst.multi:146 [INFO] found job 17063564 in provider astro proc_lsst.multi INFO: found job 17063564 in provider astro 2024-03-15 10:06:20 proc_lsst.multi:201 [INFO] cancelling 17063564 on provider astro proc_lsst.multi INFO: cancelling 17063564 on provider astro 2024-03-15 10:06:20 proc_lsst.multi:146 [INFO] found job 17063591 in provider astro proc_lsst.multi INFO: found job 17063591 in provider astro 2024-03-15 10:06:20 proc_lsst.multi:201 [INFO] cancelling 17063591 on provider astro proc_lsst.multi INFO: cancelling 17063591 on provider astro 2024-03-15 10:06:20 proc_lsst.multi:146 [INFO] found job 17063580 in provider astro proc_lsst.multi INFO: found job 17063580 in provider astro 2024-03-15 10:06:20 proc_lsst.multi:201 [INFO] cancelling 17063580 on provider astro proc_lsst.multi INFO: cancelling 17063580 on provider astro 2024-03-15 10:06:21 proc_lsst.multi:146 [INFO] found job 17063592 in provider astro proc_lsst.multi INFO: found job 17063592 in provider astro 2024-03-15 10:06:21 proc_lsst.multi:201 [INFO] cancelling 17063592 on provider astro proc_lsst.multi INFO: cancelling 17063592 on provider astro 2024-03-15 10:06:21 proc_lsst.multi:146 [INFO] found job 17063582 in provider astro proc_lsst.multi INFO: found job 17063582 in provider astro 2024-03-15 10:06:21 proc_lsst.multi:201 [INFO] cancelling 17063582 on provider astro proc_lsst.multi INFO: cancelling 17063582 on provider astro 2024-03-15 10:06:21 proc_lsst.multi:146 [INFO] found job 17063579 in provider astro proc_lsst.multi INFO: found job 17063579 in provider astro 2024-03-15 10:06:21 proc_lsst.multi:201 [INFO] cancelling 17063579 on provider astro proc_lsst.multi INFO: cancelling 17063579 on provider astro 2024-03-15 10:06:22 proc_lsst.multi:146 [INFO] found job 17063578 in provider astro proc_lsst.multi INFO: found job 17063578 in provider astro 2024-03-15 10:06:22 proc_lsst.multi:201 [INFO] cancelling 17063578 on provider astro proc_lsst.multi INFO: cancelling 17063578 on provider astro 2024-03-15 10:06:22 proc_lsst.multi:146 [INFO] found job 17063581 in provider astro proc_lsst.multi INFO: found job 17063581 in provider astro 2024-03-15 10:06:22 proc_lsst.multi:201 [INFO] cancelling 17063581 on provider astro proc_lsst.multi INFO: cancelling 17063581 on provider astro 2024-03-15 10:06:22 proc_lsst.multi:146 [INFO] found job 17063594 in provider astro proc_lsst.multi INFO: found job 17063594 in provider astro 2024-03-15 10:06:22 proc_lsst.multi:201 [INFO] cancelling 17063594 on provider astro proc_lsst.multi INFO: cancelling 17063594 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-15 10:06:22 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-15 10:06:22 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17063563', '17063564', '17063565', '17063566', '17063567', '17063568', '17063569', '17063571', '17063572', '17063573', '17063574', '17063575', '17063577', '17063578', '17063579', '17063580', '17063581', '17063582', '17063583', '17063584', '17063585', '17063586', '17063587', '17063589', '17063590', '17063591', '17063592', '17063593', '17063594', '17063595', '17063603', '17063605', '17063596', '17063597', '17063599', '17063601', '17063602', '56858'] proc_lsst.multi INFO: new jobs since last cancel ['17063563', '17063564', '17063565', '17063566', '17063567', '17063568', '17063569', '17063571', '17063572', '17063573', '17063574', '17063575', '17063577', '17063578', '17063579', '17063580', '17063581', '17063582', '17063583', '17063584', '17063585', '17063586', '17063587', '17063589', '17063590', '17063591', '17063592', '17063593', '17063594', '17063595', '17063603', '17063605', '17063596', '17063597', '17063599', '17063601', '17063602', '56858'] 2024-03-15 10:06:22 proc_lsst.multi:146 [INFO] found job 17063563 in provider astro proc_lsst.multi INFO: found job 17063563 in provider astro 2024-03-15 10:06:22 proc_lsst.multi:201 [INFO] cancelling 17063563 on provider astro proc_lsst.multi INFO: cancelling 17063563 on provider astro 2024-03-15 10:06:23 proc_lsst.multi:146 [INFO] found job 17063564 in provider astro proc_lsst.multi INFO: found job 17063564 in provider astro 2024-03-15 10:06:23 proc_lsst.multi:201 [INFO] cancelling 17063564 on provider astro proc_lsst.multi INFO: cancelling 17063564 on provider astro 2024-03-15 10:06:23 proc_lsst.multi:146 [INFO] found job 17063565 in provider astro proc_lsst.multi INFO: found job 17063565 in provider astro 2024-03-15 10:06:23 proc_lsst.multi:201 [INFO] cancelling 17063565 on provider astro proc_lsst.multi INFO: cancelling 17063565 on provider astro 2024-03-15 10:06:23 proc_lsst.multi:146 [INFO] found job 17063566 in provider astro proc_lsst.multi INFO: found job 17063566 in provider astro 2024-03-15 10:06:23 proc_lsst.multi:201 [INFO] cancelling 17063566 on provider astro proc_lsst.multi INFO: cancelling 17063566 on provider astro 2024-03-15 10:06:23 proc_lsst.multi:146 [INFO] found job 17063567 in provider astro proc_lsst.multi INFO: found job 17063567 in provider astro 2024-03-15 10:06:23 proc_lsst.multi:201 [INFO] cancelling 17063567 on provider astro proc_lsst.multi INFO: cancelling 17063567 on provider astro 2024-03-15 10:06:23 proc_lsst.multi:146 [INFO] found job 17063568 in provider astro proc_lsst.multi INFO: found job 17063568 in provider astro 2024-03-15 10:06:23 proc_lsst.multi:201 [INFO] cancelling 17063568 on provider astro proc_lsst.multi INFO: cancelling 17063568 on provider astro 2024-03-15 10:06:23 proc_lsst.multi:146 [INFO] found job 17063569 in provider astro proc_lsst.multi INFO: found job 17063569 in provider astro 2024-03-15 10:06:23 proc_lsst.multi:201 [INFO] cancelling 17063569 on provider astro proc_lsst.multi INFO: cancelling 17063569 on provider astro 2024-03-15 10:06:24 proc_lsst.multi:146 [INFO] found job 17063571 in provider astro proc_lsst.multi INFO: found job 17063571 in provider astro 2024-03-15 10:06:24 proc_lsst.multi:201 [INFO] cancelling 17063571 on provider astro proc_lsst.multi INFO: cancelling 17063571 on provider astro 2024-03-15 10:06:24 proc_lsst.multi:146 [INFO] found job 17063572 in provider astro proc_lsst.multi INFO: found job 17063572 in provider astro 2024-03-15 10:06:24 proc_lsst.multi:201 [INFO] cancelling 17063572 on provider astro proc_lsst.multi INFO: cancelling 17063572 on provider astro 2024-03-15 10:06:24 proc_lsst.multi:146 [INFO] found job 17063573 in provider astro proc_lsst.multi INFO: found job 17063573 in provider astro 2024-03-15 10:06:24 proc_lsst.multi:201 [INFO] cancelling 17063573 on provider astro proc_lsst.multi INFO: cancelling 17063573 on provider astro 2024-03-15 10:06:24 proc_lsst.multi:146 [INFO] found job 17063574 in provider astro proc_lsst.multi INFO: found job 17063574 in provider astro 2024-03-15 10:06:24 proc_lsst.multi:201 [INFO] cancelling 17063574 on provider astro proc_lsst.multi INFO: cancelling 17063574 on provider astro 2024-03-15 10:06:24 proc_lsst.multi:146 [INFO] found job 17063575 in provider astro proc_lsst.multi INFO: found job 17063575 in provider astro 2024-03-15 10:06:24 proc_lsst.multi:201 [INFO] cancelling 17063575 on provider astro proc_lsst.multi INFO: cancelling 17063575 on provider astro 2024-03-15 10:06:24 proc_lsst.multi:146 [INFO] found job 17063577 in provider astro proc_lsst.multi INFO: found job 17063577 in provider astro 2024-03-15 10:06:25 proc_lsst.multi:201 [INFO] cancelling 17063577 on provider astro proc_lsst.multi INFO: cancelling 17063577 on provider astro 2024-03-15 10:06:25 proc_lsst.multi:146 [INFO] found job 17063578 in provider astro proc_lsst.multi INFO: found job 17063578 in provider astro 2024-03-15 10:06:25 proc_lsst.multi:201 [INFO] cancelling 17063578 on provider astro proc_lsst.multi INFO: cancelling 17063578 on provider astro 2024-03-15 10:06:25 proc_lsst.multi:146 [INFO] found job 17063579 in provider astro proc_lsst.multi INFO: found job 17063579 in provider astro 2024-03-15 10:06:25 proc_lsst.multi:201 [INFO] cancelling 17063579 on provider astro proc_lsst.multi INFO: cancelling 17063579 on provider astro 2024-03-15 10:06:25 proc_lsst.multi:146 [INFO] found job 17063580 in provider astro proc_lsst.multi INFO: found job 17063580 in provider astro 2024-03-15 10:06:25 proc_lsst.multi:201 [INFO] cancelling 17063580 on provider astro proc_lsst.multi INFO: cancelling 17063580 on provider astro 2024-03-15 10:06:25 proc_lsst.multi:146 [INFO] found job 17063581 in provider astro proc_lsst.multi INFO: found job 17063581 in provider astro 2024-03-15 10:06:25 proc_lsst.multi:201 [INFO] cancelling 17063581 on provider astro proc_lsst.multi INFO: cancelling 17063581 on provider astro 2024-03-15 10:06:25 proc_lsst.multi:146 [INFO] found job 17063582 in provider astro proc_lsst.multi INFO: found job 17063582 in provider astro 2024-03-15 10:06:25 proc_lsst.multi:201 [INFO] cancelling 17063582 on provider astro proc_lsst.multi INFO: cancelling 17063582 on provider astro 2024-03-15 10:06:26 proc_lsst.multi:146 [INFO] found job 17063583 in provider astro proc_lsst.multi INFO: found job 17063583 in provider astro 2024-03-15 10:06:26 proc_lsst.multi:201 [INFO] cancelling 17063583 on provider astro proc_lsst.multi INFO: cancelling 17063583 on provider astro 2024-03-15 10:06:26 proc_lsst.multi:146 [INFO] found job 17063584 in provider astro proc_lsst.multi INFO: found job 17063584 in provider astro 2024-03-15 10:06:26 proc_lsst.multi:201 [INFO] cancelling 17063584 on provider astro proc_lsst.multi INFO: cancelling 17063584 on provider astro 2024-03-15 10:06:26 proc_lsst.multi:146 [INFO] found job 17063585 in provider astro proc_lsst.multi INFO: found job 17063585 in provider astro 2024-03-15 10:06:26 proc_lsst.multi:201 [INFO] cancelling 17063585 on provider astro proc_lsst.multi INFO: cancelling 17063585 on provider astro 2024-03-15 10:06:26 proc_lsst.multi:146 [INFO] found job 17063586 in provider astro proc_lsst.multi INFO: found job 17063586 in provider astro 2024-03-15 10:06:26 proc_lsst.multi:201 [INFO] cancelling 17063586 on provider astro proc_lsst.multi INFO: cancelling 17063586 on provider astro 2024-03-15 10:06:26 proc_lsst.multi:146 [INFO] found job 17063587 in provider astro proc_lsst.multi INFO: found job 17063587 in provider astro 2024-03-15 10:06:26 proc_lsst.multi:201 [INFO] cancelling 17063587 on provider astro proc_lsst.multi INFO: cancelling 17063587 on provider astro 2024-03-15 10:06:26 proc_lsst.multi:146 [INFO] found job 17063589 in provider astro proc_lsst.multi INFO: found job 17063589 in provider astro 2024-03-15 10:06:26 proc_lsst.multi:201 [INFO] cancelling 17063589 on provider astro proc_lsst.multi INFO: cancelling 17063589 on provider astro 2024-03-15 10:06:27 proc_lsst.multi:146 [INFO] found job 17063590 in provider astro proc_lsst.multi INFO: found job 17063590 in provider astro 2024-03-15 10:06:27 proc_lsst.multi:201 [INFO] cancelling 17063590 on provider astro proc_lsst.multi INFO: cancelling 17063590 on provider astro 2024-03-15 10:06:27 proc_lsst.multi:146 [INFO] found job 17063591 in provider astro proc_lsst.multi INFO: found job 17063591 in provider astro 2024-03-15 10:06:27 proc_lsst.multi:201 [INFO] cancelling 17063591 on provider astro proc_lsst.multi INFO: cancelling 17063591 on provider astro 2024-03-15 10:06:27 proc_lsst.multi:146 [INFO] found job 17063592 in provider astro proc_lsst.multi INFO: found job 17063592 in provider astro 2024-03-15 10:06:27 proc_lsst.multi:201 [INFO] cancelling 17063592 on provider astro proc_lsst.multi INFO: cancelling 17063592 on provider astro 2024-03-15 10:06:27 proc_lsst.multi:146 [INFO] found job 17063593 in provider astro proc_lsst.multi INFO: found job 17063593 in provider astro 2024-03-15 10:06:27 proc_lsst.multi:201 [INFO] cancelling 17063593 on provider astro proc_lsst.multi INFO: cancelling 17063593 on provider astro 2024-03-15 10:06:27 proc_lsst.multi:146 [INFO] found job 17063594 in provider astro proc_lsst.multi INFO: found job 17063594 in provider astro 2024-03-15 10:06:27 proc_lsst.multi:201 [INFO] cancelling 17063594 on provider astro proc_lsst.multi INFO: cancelling 17063594 on provider astro 2024-03-15 10:06:27 proc_lsst.multi:146 [INFO] found job 17063595 in provider astro proc_lsst.multi INFO: found job 17063595 in provider astro 2024-03-15 10:06:27 proc_lsst.multi:201 [INFO] cancelling 17063595 on provider astro proc_lsst.multi INFO: cancelling 17063595 on provider astro 2024-03-15 10:06:28 proc_lsst.multi:146 [INFO] found job 17063603 in provider escience_a40 proc_lsst.multi INFO: found job 17063603 in provider escience_a40 2024-03-15 10:06:28 proc_lsst.multi:201 [INFO] cancelling 17063603 on provider escience_a40 proc_lsst.multi INFO: cancelling 17063603 on provider escience_a40 2024-03-15 10:06:28 proc_lsst.multi:146 [INFO] found job 17063605 in provider escience_a40 proc_lsst.multi INFO: found job 17063605 in provider escience_a40 2024-03-15 10:06:28 proc_lsst.multi:201 [INFO] cancelling 17063605 on provider escience_a40 proc_lsst.multi INFO: cancelling 17063605 on provider escience_a40 2024-03-15 10:06:28 proc_lsst.multi:146 [INFO] found job 17063596 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063596 in provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:201 [INFO] cancelling 17063596 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063596 on provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:146 [INFO] found job 17063597 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063597 in provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:201 [INFO] cancelling 17063597 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063597 on provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:146 [INFO] found job 17063599 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063599 in provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:201 [INFO] cancelling 17063599 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063599 on provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:146 [INFO] found job 17063601 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063601 in provider escience_rtx6k 2024-03-15 10:06:28 proc_lsst.multi:201 [INFO] cancelling 17063601 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063601 on provider escience_rtx6k 2024-03-15 10:06:29 proc_lsst.multi:146 [INFO] found job 17063602 in provider escience_rtx6k proc_lsst.multi INFO: found job 17063602 in provider escience_rtx6k 2024-03-15 10:06:29 proc_lsst.multi:201 [INFO] cancelling 17063602 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17063602 on provider escience_rtx6k 2024-03-15 10:06:29 proc_lsst.multi:146 [INFO] found job 56858 in provider local proc_lsst.multi INFO: found job 56858 in provider local 2024-03-15 10:06:29 proc_lsst.multi:201 [INFO] cancelling 56858 on provider local proc_lsst.multi INFO: cancelling 56858 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 56858 parsl.providers.local.local WARNING: Failed to kill PID: 56858 and child processes on local 2024-03-15 10:06:30 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 170.4662 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210510_A1c_science#step0_20240315T100328Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 170.4744 seconds; current memory usage: 0.283 Gibyte, delta: 0.009 Gibyte, peak delta: 0.011 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 181.3985 seconds; current memory usage: 0.283 Gibyte, delta: 0.146 Gibyte, peak delta: 0.148 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.285 Gibyte (main), 0.285 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210510_A1c_science#step0_20240315T100328Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 3m3.396s user 0m32.903s sys 0m15.054s