+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190504/A1c --output-run DEEP/20190504/A1c/science#step0/20240308T100021Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.0649 seconds; current memory usage: 0.179 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0043 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0091 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/pipeline_orig.qgraph DEEP/20190504/A1c/science#step0/20240308T100021Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.9408 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 8.1284 seconds; current memory usage: 0.280 Gibyte, delta: 0.101 Gibyte, peak delta: 0.090 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.280 Gibyte (main), 0.274 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0213 seconds; current memory usage: 0.280 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.280 Gibyte (main), 0.274 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 930 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190504_A1c_science#step0_20240308T100021Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1816 seconds; current memory usage: 0.280 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.280 Gibyte (main), 0.274 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 932 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.3126 seconds; current memory usage: 0.282 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.282 Gibyte (main), 0.274 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: dcf1ce7c-1bce-4e3c-8516-1da56e08aef5 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-08 10:00:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-08 10:00:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 10:00:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-08 10:00:46 proc_lsst.multi:170 [INFO] [multi] job_id 30716 proc_lsst.multi INFO: [multi] job_id 30716 2024-03-08 10:00:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:46 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 30716 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 30716 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 30716 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash af3624e88948c922dd6b79f9e2cbfc07 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd56d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7d10a525-03f1-4ba9-af1c-737542e95a97_isrForCrosstalkSources_855237_12.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7d10a525-03f1-4ba9-af1c-737542e95a97_isrForCrosstalkSources_855237_12.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 024ca0efd179061c73574679d023403b parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a4579e50-863e-48ec-a408-387570909876_isrForCrosstalkSources_855236_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a4579e50-863e-48ec-a408-387570909876_isrForCrosstalkSources_855236_47.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 30a2cf08f035df85367f18b9afe31ad2 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/84b56d94-7d18-4e3e-aa47-2eb3fbf155f1_isrForCrosstalkSources_855237_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/84b56d94-7d18-4e3e-aa47-2eb3fbf155f1_isrForCrosstalkSources_855237_13.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 2611dd1a40957f5e6333443303f878a7 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f07ef528-72db-4507-868e-7d2f86241f90_isrForCrosstalkSources_855156_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f07ef528-72db-4507-868e-7d2f86241f90_isrForCrosstalkSources_855156_55.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash e7e6905540f3f2d059fc08ea300c5060 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd56480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9cb8593b-497e-4cc7-816f-6b4f5e2419ab_isrForCrosstalkSources_855233_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9cb8593b-497e-4cc7-816f-6b4f5e2419ab_isrForCrosstalkSources_855233_20.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 9b5dec039f47ea919a9906969f6b3ccc parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd574c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/76bd26a1-1893-4ee0-82a6-bddbd19c80ee_isrForCrosstalkSources_855232_32.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/76bd26a1-1893-4ee0-82a6-bddbd19c80ee_isrForCrosstalkSources_855232_32.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash c172512dca8b331f102376e833e0cae9 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5120dff6-fa56-4dd3-a923-d8d2a6654166_isrForCrosstalkSources_855156_29.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5120dff6-fa56-4dd3-a923-d8d2a6654166_isrForCrosstalkSources_855156_29.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 2ca11a2b6b25e0a4313f23461d39d607 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5cad50eb-bcc8-42ae-9350-9edec3936a91_isrForCrosstalkSources_855236_49.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5cad50eb-bcc8-42ae-9350-9edec3936a91_isrForCrosstalkSources_855236_49.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash aaf2d52c6a230d798307489f66d88b62 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd56660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/31305552-509e-48fa-888c-1d3674bc283e_isrForCrosstalkSources_855233_62.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/31305552-509e-48fa-888c-1d3674bc283e_isrForCrosstalkSources_855233_62.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 9a626c4c8d74eab2081d5345540bb06c parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/cf299999-c443-4478-9e1c-1fa1d53bdc2c_isrForCrosstalkSources_855242_56.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/cf299999-c443-4478-9e1c-1fa1d53bdc2c_isrForCrosstalkSources_855242_56.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash a37a58edcf10d6ec99db54456777fa74 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd56f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/585a4569-84f3-4a96-b4e9-b7ab0cd1f35f_isrForCrosstalkSources_855237_51.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/585a4569-84f3-4a96-b4e9-b7ab0cd1f35f_isrForCrosstalkSources_855237_51.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash fb881a77c66759b3522c1313cebe3f7e parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/53647791-1a75-4ff2-82f9-1248aeb08086_isrForCrosstalkSources_855234_37.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/53647791-1a75-4ff2-82f9-1248aeb08086_isrForCrosstalkSources_855234_37.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 6c07d32ad6ff70f54d42a0b9520041c9 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd579c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/629ff215-4e1f-4a8a-aec2-34aa09e46bc3_isrForCrosstalkSources_855240_43.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/629ff215-4e1f-4a8a-aec2-34aa09e46bc3_isrForCrosstalkSources_855240_43.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 1a2177476854818613bbbe8de941c361 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5e998b9e-4555-4708-a9a8-7ce6c31897d6_isrForCrosstalkSources_855234_33.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5e998b9e-4555-4708-a9a8-7ce6c31897d6_isrForCrosstalkSources_855234_33.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 0445ab17d37bba4dac3176abea97714d parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d859edb6-6557-4ebe-a0f4-2f4cf9304a07_isrForCrosstalkSources_855154_38.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d859edb6-6557-4ebe-a0f4-2f4cf9304a07_isrForCrosstalkSources_855154_38.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 01ee273b4525897ab2b96ced976015ec parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cb7e334a-4c0c-40aa-9c33-e79a2612e2ab_isrForCrosstalkSources_855241_47.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cb7e334a-4c0c-40aa-9c33-e79a2612e2ab_isrForCrosstalkSources_855241_47.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 9e59b35245c1818d5e2c3dedd71751a7 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/848be6ef-c7e9-4658-a8eb-e7638fd87d4f_isrForCrosstalkSources_855232_14.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/848be6ef-c7e9-4658-a8eb-e7638fd87d4f_isrForCrosstalkSources_855232_14.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash b82e56dfc2bf8b7e272bd0f52501c4ed parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/53077817-ff60-4e7b-9ba2-159266d69c46_isrForCrosstalkSources_855238_54.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/53077817-ff60-4e7b-9ba2-159266d69c46_isrForCrosstalkSources_855238_54.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash fd17528b21100b410e97ebe77bd6168c parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/676fecaf-41a1-4b9f-aebb-40c267f13323_isrForCrosstalkSources_855240_38.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/676fecaf-41a1-4b9f-aebb-40c267f13323_isrForCrosstalkSources_855240_38.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash aee0faed429ea9325ebe4436fee6d3be parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bd55e3e5-99e5-4b99-a3b3-ee906cb7ab3f_isrForCrosstalkSources_855236_20.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bd55e3e5-99e5-4b99-a3b3-ee906cb7ab3f_isrForCrosstalkSources_855236_20.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash fbf31e56988e5aa522a99cca5e02e29b parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/508a58ce-94b5-4ec1-a9f8-70ed09ed2a1b_isrForCrosstalkSources_855237_26.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/508a58ce-94b5-4ec1-a9f8-70ed09ed2a1b_isrForCrosstalkSources_855237_26.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 0edafa04d36f33a1c2516ff767fbb998 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8446130-c7fd-4171-afcb-30fa7bf8347f_isrForCrosstalkSources_855155_21.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8446130-c7fd-4171-afcb-30fa7bf8347f_isrForCrosstalkSources_855155_21.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 4d9b8d1508a47d437955788798bec67f parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9f298d1-10c9-4a53-8e91-458555c22574_isrForCrosstalkSources_855231_4.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9f298d1-10c9-4a53-8e91-458555c22574_isrForCrosstalkSources_855231_4.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 936b4e353e1b75cf24365d0eda91f0b2 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/980ebe94-d071-42a7-a6c6-06b59b8752db_isrForCrosstalkSources_855240_28.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/980ebe94-d071-42a7-a6c6-06b59b8752db_isrForCrosstalkSources_855240_28.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 5e7c40e8fe09ac43fca80b89a49532ad parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/47ba14dc-bf3a-4a17-bbd1-db78fb1d2f70_isrForCrosstalkSources_855233_55.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/47ba14dc-bf3a-4a17-bbd1-db78fb1d2f70_isrForCrosstalkSources_855233_55.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 67b18df727ea919a608195d96f905270 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/133c84e7-8363-444b-9bed-8eb9d97d25de_isrForCrosstalkSources_855235_26.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/133c84e7-8363-444b-9bed-8eb9d97d25de_isrForCrosstalkSources_855235_26.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash f8a9fd8a966bff9a8f34a963ff8e7a54 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/a7b0ead8-6164-4037-8a69-36978ed2f65d_isrForCrosstalkSources_855241_33.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/a7b0ead8-6164-4037-8a69-36978ed2f65d_isrForCrosstalkSources_855241_33.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 741c1e90b8a3cf7a08e9b92d291537de parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7d45c4a6-93c6-4218-80d3-8ae959ae6763_isrForCrosstalkSources_855234_26.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7d45c4a6-93c6-4218-80d3-8ae959ae6763_isrForCrosstalkSources_855234_26.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 23194809df74c85b2b3f49a4e4c27757 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/227654c3-21e6-4857-b815-99369c113585_isrForCrosstalkSources_855233_38.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/227654c3-21e6-4857-b815-99369c113585_isrForCrosstalkSources_855233_38.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 985279c79a357b06d38c8b739edbe6f9 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/72f27eb5-1ec9-43e7-8e52-82da0373a0f4_isrForCrosstalkSources_855238_37.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/72f27eb5-1ec9-43e7-8e52-82da0373a0f4_isrForCrosstalkSources_855238_37.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 8120c5a05ad551dcc1f188cffbb34d0c parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c6a31c6c-77d9-4bef-a302-35e2057688a0_isrForCrosstalkSources_855242_38.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c6a31c6c-77d9-4bef-a302-35e2057688a0_isrForCrosstalkSources_855242_38.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 7c5f9832a3628be7527040bd718c08ea parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6f9baa4b-e779-4cc1-8cbc-f964a0adb228_isrForCrosstalkSources_855241_20.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6f9baa4b-e779-4cc1-8cbc-f964a0adb228_isrForCrosstalkSources_855241_20.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash d3b2b59a6fbd454fc7ff4f186fa15f83 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0dd4a5d4-9302-4a37-af61-5293233f4e90_isrForCrosstalkSources_855156_34.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0dd4a5d4-9302-4a37-af61-5293233f4e90_isrForCrosstalkSources_855156_34.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash a4e7ec2ee5b25cbe1f26c4b8fd4dc642 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8ad96fda-8ed1-4c28-807e-a8463d290eec_isrForCrosstalkSources_855233_54.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8ad96fda-8ed1-4c28-807e-a8463d290eec_isrForCrosstalkSources_855233_54.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 9b9e68efd9a079eb3fc48c11f2a37ca6 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9421d547-733f-48d7-acf9-eb3154a5a978_isrForCrosstalkSources_855240_56.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9421d547-733f-48d7-acf9-eb3154a5a978_isrForCrosstalkSources_855240_56.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash a3e5d645c846570a5fb97b2e727b1aa5 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f91041a5-6d40-486e-8d37-eb5b0fc674b0_isrForCrosstalkSources_855241_24.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f91041a5-6d40-486e-8d37-eb5b0fc674b0_isrForCrosstalkSources_855241_24.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash c33be7ce417c20ad93b5d0bc68b62543 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ab951d3c-e8d5-4c10-9c05-a8683f5bb7b4_isrForCrosstalkSources_855242_20.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ab951d3c-e8d5-4c10-9c05-a8683f5bb7b4_isrForCrosstalkSources_855242_20.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash d236cdf831ef06ad2ac8ba142e5eabff parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/cc3155cf-a48b-461e-8934-b4a0b3e3af3c_isrForCrosstalkSources_855236_22.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/cc3155cf-a48b-461e-8934-b4a0b3e3af3c_isrForCrosstalkSources_855236_22.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash c0c4cc7a73956fbc93400c8d91bbd8b5 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2e15e4ff-725d-4cb9-9886-a0c4e7a07a8e_isrForCrosstalkSources_855238_7.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2e15e4ff-725d-4cb9-9886-a0c4e7a07a8e_isrForCrosstalkSources_855238_7.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 80143c8ab79b51f178aef288add1d722 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/31e46ff2-767c-49e2-a793-afdbc6904947_isrForCrosstalkSources_855242_11.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/31e46ff2-767c-49e2-a793-afdbc6904947_isrForCrosstalkSources_855242_11.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash e513ef24f4e58c910b2fcefb8266a61e parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0fd69046-8072-4ab2-ae05-e22329f672cf_isrForCrosstalkSources_855231_53.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0fd69046-8072-4ab2-ae05-e22329f672cf_isrForCrosstalkSources_855231_53.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash a8ae115a417ba0badb73c67fb74dff4a parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bd11c6bd-d298-4a51-86cb-ae61eba185ae_isrForCrosstalkSources_855240_48.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bd11c6bd-d298-4a51-86cb-ae61eba185ae_isrForCrosstalkSources_855240_48.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash a9baf3a58904d7ad00dc75a22a551b24 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93f965db-cb41-423b-9739-f96ddcc92e6a_isrForCrosstalkSources_855155_52.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93f965db-cb41-423b-9739-f96ddcc92e6a_isrForCrosstalkSources_855155_52.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 3da8a38bca2250c46989698a4b9ceb65 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/946fd4b9-97b0-429d-8d5b-9e27db30435c_isrForCrosstalkSources_855236_52.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/946fd4b9-97b0-429d-8d5b-9e27db30435c_isrForCrosstalkSources_855236_52.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash cae465c55922b584c35d2f40a5132a8d parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/ae8ae8b5-f6fd-4165-925b-9dbabbd42515_isrForCrosstalkSources_855232_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/ae8ae8b5-f6fd-4165-925b-9dbabbd42515_isrForCrosstalkSources_855232_48.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash d019f6f76d73ebb137c39a4a0c45b7f7 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aafe40dd-d5be-4c95-89bb-3f59de8b3b9b_isrForCrosstalkSources_855233_57.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aafe40dd-d5be-4c95-89bb-3f59de8b3b9b_isrForCrosstalkSources_855233_57.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash ea09934e841537b655528e7d3fdae735 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/78ed1b05-8cf5-48e8-bafa-316e79570910_isrForCrosstalkSources_855241_25.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/78ed1b05-8cf5-48e8-bafa-316e79570910_isrForCrosstalkSources_855241_25.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 66fe6e27e57920ee6435ae81e75dc8a2 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/fd49db3f-a198-4041-b977-5ca699e9962e_isrForCrosstalkSources_855236_17.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/fd49db3f-a198-4041-b977-5ca699e9962e_isrForCrosstalkSources_855236_17.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 576266cd0ffe66331c89747710519def parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5de965d7-68bd-4e3c-9581-04fafc935684_isrForCrosstalkSources_855233_16.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5de965d7-68bd-4e3c-9581-04fafc935684_isrForCrosstalkSources_855233_16.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash b55111042f9c3acb2f92bcd534d88da5 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c5f85583-04b3-4d46-b357-b3da66ee461b_isrForCrosstalkSources_855155_36.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c5f85583-04b3-4d46-b357-b3da66ee461b_isrForCrosstalkSources_855155_36.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash ca9da9115232d844e24b3782bfba6203 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/429c1b10-c35b-4483-b36f-baa7af2b7c55_isrForCrosstalkSources_855241_29.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/429c1b10-c35b-4483-b36f-baa7af2b7c55_isrForCrosstalkSources_855241_29.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 1606c451c95e3a0c60891867e5e86406 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/482db7ab-ca8c-4190-86b9-f8b74fcbdbed_isrForCrosstalkSources_855237_32.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/482db7ab-ca8c-4190-86b9-f8b74fcbdbed_isrForCrosstalkSources_855237_32.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash b9e2b02592bfa9f518e299d60faa41d7 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a4e2fdf7-6838-4401-b138-ba0f80ddb053_isrForCrosstalkSources_855234_60.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a4e2fdf7-6838-4401-b138-ba0f80ddb053_isrForCrosstalkSources_855234_60.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash cdd78c672e56379de70199e704bc44ff parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de3251e2-67e5-48da-b4d4-5ed34fc9497c_isrForCrosstalkSources_855233_14.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de3251e2-67e5-48da-b4d4-5ed34fc9497c_isrForCrosstalkSources_855233_14.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 0eb10d1fe14e29c8eabfb50f1f1f0fc4 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ae4c54b-8880-45a4-b04e-6ce98e25fb5f_isrForCrosstalkSources_855234_41.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ae4c54b-8880-45a4-b04e-6ce98e25fb5f_isrForCrosstalkSources_855234_41.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 48e8eeae26319efe9f091b73898539ec parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/87d2a957-4aff-46b6-87b3-29bb39fd52ba_isrForCrosstalkSources_855242_5.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/87d2a957-4aff-46b6-87b3-29bb39fd52ba_isrForCrosstalkSources_855242_5.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 2c2342cf4c58d39aa778baac9a641190 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/43d56360-5024-4e89-95d3-3e62830f8439_isrForCrosstalkSources_855238_30.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/43d56360-5024-4e89-95d3-3e62830f8439_isrForCrosstalkSources_855238_30.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 4e613d9161dc109daa4cf2159a3338a9 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/50dc01e8-7fc6-4e92-b899-dc4a154e63ee_isrForCrosstalkSources_855237_33.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/50dc01e8-7fc6-4e92-b899-dc4a154e63ee_isrForCrosstalkSources_855237_33.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash bb632ded6b7509e1a17fa92227b7e0ae parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/01203b56-0f1a-4fe1-988e-4d8f9532918c_isrForCrosstalkSources_855237_57.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/01203b56-0f1a-4fe1-988e-4d8f9532918c_isrForCrosstalkSources_855237_57.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 2d844dc6f28d63c8aed7dc8ef55c7058 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/7be63ae7-90e0-4cd7-9bc0-fc25f62469bf_isrForCrosstalkSources_855156_39.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/7be63ae7-90e0-4cd7-9bc0-fc25f62469bf_isrForCrosstalkSources_855156_39.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 43e300156604297aeca33e22d3f58384 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/41d81427-5004-4240-8d95-d3e7525c5687_isrForCrosstalkSources_855240_57.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/41d81427-5004-4240-8d95-d3e7525c5687_isrForCrosstalkSources_855240_57.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 6e70e5a7f1f8d61ebffe55a68527e575 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c8dd9e81-1b43-451d-b4e0-9097d653f77d_isrForCrosstalkSources_855232_21.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c8dd9e81-1b43-451d-b4e0-9097d653f77d_isrForCrosstalkSources_855232_21.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 90f82c236aac6dfe27c68fd0c01fbc87 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/547c98fe-ac5b-442b-a23e-e8ecf8745f2b_isrForCrosstalkSources_855242_10.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/547c98fe-ac5b-442b-a23e-e8ecf8745f2b_isrForCrosstalkSources_855242_10.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 8e380508ffee4e2eb93877cc93227403 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2f117df-4f7b-4d83-a223-8711d6704c55_isrForCrosstalkSources_855155_29.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2f117df-4f7b-4d83-a223-8711d6704c55_isrForCrosstalkSources_855155_29.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 33d6fac2a2796cfb4eecfe9891bbfea9 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7e8043d4-832d-4016-97e1-ce48a7ad47c6_isrForCrosstalkSources_855236_53.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7e8043d4-832d-4016-97e1-ce48a7ad47c6_isrForCrosstalkSources_855236_53.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 3b9cd532a1d470aeb5c18bd3864e9f4d parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24acabc7-98ff-489b-8d09-63d1f51143bb_isrForCrosstalkSources_855241_41.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24acabc7-98ff-489b-8d09-63d1f51143bb_isrForCrosstalkSources_855241_41.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 29fd90b164db26dfeba4fa87e4c32d6b parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d2f344e-996e-4ea4-b139-47482055f379_isrForCrosstalkSources_855237_50.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d2f344e-996e-4ea4-b139-47482055f379_isrForCrosstalkSources_855237_50.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 917805c8298bc858d9df1be483019a90 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/95d2d6a3-9f9a-4a27-8d10-4b7d81948f44_isrForCrosstalkSources_855235_36.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/95d2d6a3-9f9a-4a27-8d10-4b7d81948f44_isrForCrosstalkSources_855235_36.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 2c9eb2489a5e361b871503cf9db60810 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24149a6f-2557-4cc7-a771-f67fc5463261_isrForCrosstalkSources_855231_59.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24149a6f-2557-4cc7-a771-f67fc5463261_isrForCrosstalkSources_855231_59.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 44d86a05d27eb4e187329fdaabedc32e parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c04ad081-122c-4c14-ba8a-bd4816052fd5_isrForCrosstalkSources_855237_40.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c04ad081-122c-4c14-ba8a-bd4816052fd5_isrForCrosstalkSources_855237_40.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash c8a8d7b67b86be22c68f71c36bf36073 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e4796a41-1666-4920-91c8-5c6e2287776d_isrForCrosstalkSources_855235_8.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e4796a41-1666-4920-91c8-5c6e2287776d_isrForCrosstalkSources_855235_8.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash dacb519722b2920aa6b9a68b3a960079 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6bad174b-4671-4f47-8823-962fc2a6de56_isrForCrosstalkSources_855241_8.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6bad174b-4671-4f47-8823-962fc2a6de56_isrForCrosstalkSources_855241_8.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 328cd57021352387d2e04f8d13502849 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f2ca7c97-9e9f-4d63-a6c9-56e441dfc55a_isrForCrosstalkSources_855241_43.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f2ca7c97-9e9f-4d63-a6c9-56e441dfc55a_isrForCrosstalkSources_855241_43.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 9045b159299b659395bdc6c298058065 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/e11a38b3-97be-4386-aa6a-95993bb43c49_isrForCrosstalkSources_855233_37.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/e11a38b3-97be-4386-aa6a-95993bb43c49_isrForCrosstalkSources_855233_37.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 3d4c09b305335157966922d66b2fb9d3 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/67a88562-91d6-44cf-b739-eeea56adba21_isrForCrosstalkSources_855234_52.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/67a88562-91d6-44cf-b739-eeea56adba21_isrForCrosstalkSources_855234_52.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash b9bdc7f709d701511a6ba2fbc8503136 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/0ffb7868-4142-46db-b05d-c8b8376561d9_isrForCrosstalkSources_855235_29.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/0ffb7868-4142-46db-b05d-c8b8376561d9_isrForCrosstalkSources_855235_29.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash ebd82d95112a3cee46d80657271fe70a parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c2238409-0a02-4056-83ca-b63717ab45d7_isrForCrosstalkSources_855236_4.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c2238409-0a02-4056-83ca-b63717ab45d7_isrForCrosstalkSources_855236_4.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash b05d6a283675618d3ee79a784ba16495 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/097dab31-bd07-480b-b08a-028ee1df885b_isrForCrosstalkSources_855239_5.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/097dab31-bd07-480b-b08a-028ee1df885b_isrForCrosstalkSources_855239_5.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash ba7d685d9ad290e489887fb37563fb19 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2927f888-c00f-4f0c-9754-f065d5cd984f_isrForCrosstalkSources_855238_55.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2927f888-c00f-4f0c-9754-f065d5cd984f_isrForCrosstalkSources_855238_55.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 7314b5f115ca6b1c8af2c5be5eeb8cf7 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8c77b052-22d3-4621-a84e-7262e1fa5f8f_isrForCrosstalkSources_855233_53.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8c77b052-22d3-4621-a84e-7262e1fa5f8f_isrForCrosstalkSources_855233_53.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash c33056be7f228fec8ae1c6f3cb5b0b13 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3e0754d0-967f-4cd9-92b1-5fa9adeb19e3_isrForCrosstalkSources_855156_33.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3e0754d0-967f-4cd9-92b1-5fa9adeb19e3_isrForCrosstalkSources_855156_33.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash b844f41125a084ec3b8e24f31df69fcb parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e293711e-dcd0-41fa-9276-258efbbe40b4_isrForCrosstalkSources_855235_51.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e293711e-dcd0-41fa-9276-258efbbe40b4_isrForCrosstalkSources_855235_51.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash eaccf7c1d0b980a26badb18f3edc2636 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5dbfd987-a196-40e4-bdd1-13bc7f696985_isrForCrosstalkSources_855242_47.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5dbfd987-a196-40e4-bdd1-13bc7f696985_isrForCrosstalkSources_855242_47.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 56f925546ede14eaf18c1138128d2248 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/698a20fe-667e-4c21-bd80-434919751799_isrForCrosstalkSources_855240_27.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/698a20fe-667e-4c21-bd80-434919751799_isrForCrosstalkSources_855240_27.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash ced37790ea6bc76bc9f7f29b78901ae3 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fe980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a6f1a15-62e4-4a1a-b714-32df2bff67d8_isrForCrosstalkSources_855235_30.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a6f1a15-62e4-4a1a-b714-32df2bff67d8_isrForCrosstalkSources_855235_30.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash abe9258094a7fe9338947be2eb984820 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/05b0656d-081d-4661-ac23-8af9ac1d5db8_isrForCrosstalkSources_855234_16.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/05b0656d-081d-4661-ac23-8af9ac1d5db8_isrForCrosstalkSources_855234_16.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 8ffefea3857830376dab9adacd4f0806 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6feac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/87f4d9ac-cd86-497b-b15e-a9c230424cc8_isrForCrosstalkSources_855240_41.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/87f4d9ac-cd86-497b-b15e-a9c230424cc8_isrForCrosstalkSources_855240_41.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 35769237b2f2b8999143dfda0576f8e2 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/cad6ab34-38ec-4544-8caa-e6d0b81f3e71_isrForCrosstalkSources_855238_1.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/cad6ab34-38ec-4544-8caa-e6d0b81f3e71_isrForCrosstalkSources_855238_1.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash c5c2306478f8aa92768dc3449f810f6b parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6feb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/de7f1d17-7825-467f-b440-4932c5b4bc94_isrForCrosstalkSources_855234_24.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/de7f1d17-7825-467f-b440-4932c5b4bc94_isrForCrosstalkSources_855234_24.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 83ff54d6d70d4b2c45fe2964120259a5 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6feca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/44ae3daa-8271-4e27-be47-2cee65d46f78_isrForCrosstalkSources_855242_61.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/44ae3daa-8271-4e27-be47-2cee65d46f78_isrForCrosstalkSources_855242_61.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 01972f7f11b286b6e1fd5c0ff48ae99b parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/4050e335-f406-4032-83c3-9bf7220bcf26_isrForCrosstalkSources_855241_13.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/4050e335-f406-4032-83c3-9bf7220bcf26_isrForCrosstalkSources_855241_13.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash bc8f4e40b0e484da7f786b901c37e8cb parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/1cdece92-f054-430e-bbbc-c18276d6904e_isrForCrosstalkSources_855233_29.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/1cdece92-f054-430e-bbbc-c18276d6904e_isrForCrosstalkSources_855233_29.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash ecb4ff51019ca34dc17bac7dcd39fc13 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/486cd0d5-71ed-4781-a4cb-c80f68463cae_isrForCrosstalkSources_855241_55.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/486cd0d5-71ed-4781-a4cb-c80f68463cae_isrForCrosstalkSources_855241_55.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 94aa0cd9bb8e789a079290b731b55d8c parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/518b2b82-66c0-4f43-877e-03e03ce94ea4_isrForCrosstalkSources_855231_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/518b2b82-66c0-4f43-877e-03e03ce94ea4_isrForCrosstalkSources_855231_23.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash a7f3b8ed276ec5cd13f644de8cacab39 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f55e0944-391b-408d-8098-a7ab7536c42c_isrForCrosstalkSources_855241_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f55e0944-391b-408d-8098-a7ab7536c42c_isrForCrosstalkSources_855241_30.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash c5c2b03b3cbe0de39eb737af84d762eb parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/de8d3c08-fe38-4d4d-b267-4232b2c99632_isrForCrosstalkSources_855239_17.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/de8d3c08-fe38-4d4d-b267-4232b2c99632_isrForCrosstalkSources_855239_17.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 3365b2c6fe9ccfc9a38a0e809a5fc79c parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/54c19af1-5e66-4756-a9ce-68d5cd874e14_isrForCrosstalkSources_855240_51.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/54c19af1-5e66-4756-a9ce-68d5cd874e14_isrForCrosstalkSources_855240_51.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash aeac05c01580cd5ac4e6fc1ef666ff02 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b9148068-2fd2-4b46-b000-eea00b5dfe7d_isrForCrosstalkSources_855154_5.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b9148068-2fd2-4b46-b000-eea00b5dfe7d_isrForCrosstalkSources_855154_5.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 16e379e52d876e30c783053c715125da parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/81634672-243f-4bd8-b494-316f94c361fb_isrForCrosstalkSources_855235_12.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/81634672-243f-4bd8-b494-316f94c361fb_isrForCrosstalkSources_855235_12.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 223fd19802ebf9005c1697cf72525ff4 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fed7fff3-7507-4eea-8c0a-ae5726dea14f_isrForCrosstalkSources_855238_2.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fed7fff3-7507-4eea-8c0a-ae5726dea14f_isrForCrosstalkSources_855238_2.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash f910909c8d653010de3b4ee82c664ab0 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6eba6b60-26d8-4d54-98b4-1b364194be27_isrForCrosstalkSources_855240_11.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6eba6b60-26d8-4d54-98b4-1b364194be27_isrForCrosstalkSources_855240_11.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash f870cf909bf20b9b94810010d2f34cf5 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/25fc0f29-ecf3-46e2-b7f6-f6616b6f21cb_isrForCrosstalkSources_855242_35.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/25fc0f29-ecf3-46e2-b7f6-f6616b6f21cb_isrForCrosstalkSources_855242_35.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 16589d2b7c45418767718ca75c2383de parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/de2ae44f-e527-4548-b701-4c8e25f6e66e_isrForCrosstalkSources_855231_5.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/de2ae44f-e527-4548-b701-4c8e25f6e66e_isrForCrosstalkSources_855231_5.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 5615bcea4d20eabf08551f982e83773a parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/64b5cfca-7b34-4cb3-9226-9133bf329b38_isrForCrosstalkSources_855240_21.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/64b5cfca-7b34-4cb3-9226-9133bf329b38_isrForCrosstalkSources_855240_21.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash c2f02f1c16358a128de8e437b1801fb0 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/dc5e1853-6761-4713-ada1-7ac3ad018019_isrForCrosstalkSources_855155_9.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/dc5e1853-6761-4713-ada1-7ac3ad018019_isrForCrosstalkSources_855155_9.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 09390f9e9d2647cba85fa5cc01669617 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/66b414b3-6ba4-4f8a-8e67-d29d6c8fc481_isrForCrosstalkSources_855235_4.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/66b414b3-6ba4-4f8a-8e67-d29d6c8fc481_isrForCrosstalkSources_855235_4.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash d0be855e45b5b55d052c00586ccde49f parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/9f120a28-32ce-4c7d-a1eb-cf9f69aa14db_isrForCrosstalkSources_855234_12.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/9f120a28-32ce-4c7d-a1eb-cf9f69aa14db_isrForCrosstalkSources_855234_12.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 1b3c238d405898c5f0f62e662a180453 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/381da02e-6142-40f2-bb70-52d67030094d_isrForCrosstalkSources_855236_58.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/381da02e-6142-40f2-bb70-52d67030094d_isrForCrosstalkSources_855236_58.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 863ce75fd775a8e4b6500b3d43f6f3df parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d81d5922-9f1f-446b-b854-bac768ff0d5f_isrForCrosstalkSources_855155_32.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d81d5922-9f1f-446b-b854-bac768ff0d5f_isrForCrosstalkSources_855155_32.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 4ca04508a7448ba76dfc1cfc729cf147 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3c03354c-39ae-444a-aaa1-5b3353edce8e_isrForCrosstalkSources_855233_36.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3c03354c-39ae-444a-aaa1-5b3353edce8e_isrForCrosstalkSources_855233_36.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 12af1fffec0658ed9c169d437ba0a5a5 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ff9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2bbc63f3-0a59-4889-9a28-723db80cd4a0_isrForCrosstalkSources_855238_62.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2bbc63f3-0a59-4889-9a28-723db80cd4a0_isrForCrosstalkSources_855238_62.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 97faa5e1c80fc50bd1ad506291232c06 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b06c0a6e-76c7-4545-8d29-bfb651c1c6a5_isrForCrosstalkSources_855241_27.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b06c0a6e-76c7-4545-8d29-bfb651c1c6a5_isrForCrosstalkSources_855241_27.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 84f62a5e934a8dde395f3e4e22693eac parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/19831123-bc35-4b2b-affa-3f4ec146079b_isrForCrosstalkSources_855240_13.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/19831123-bc35-4b2b-affa-3f4ec146079b_isrForCrosstalkSources_855240_13.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 7ba2b9382a71914ec49587c124ba9689 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d5024958-a02a-430b-9ce5-616f1fb5406c_isrForCrosstalkSources_855238_49.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d5024958-a02a-430b-9ce5-616f1fb5406c_isrForCrosstalkSources_855238_49.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 9b6b215b14c0ab18b7273cee45df2b2e parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9c9ac09a-5851-4bea-a05f-76918a177984_isrForCrosstalkSources_855232_57.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9c9ac09a-5851-4bea-a05f-76918a177984_isrForCrosstalkSources_855232_57.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 2d46435b05916c2bf05e6c70163f01eb parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/e5c5eb98-5a73-4619-a7c4-1129f74c3801_isrForCrosstalkSources_855240_58.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/e5c5eb98-5a73-4619-a7c4-1129f74c3801_isrForCrosstalkSources_855240_58.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 3d243587047603af0dcca70820a59f45 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5eea1dd5-ad36-47ea-b4be-b2657365e99d_isrForCrosstalkSources_855238_47.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5eea1dd5-ad36-47ea-b4be-b2657365e99d_isrForCrosstalkSources_855238_47.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 10674cc85e75b572584898a9e1eabcf5 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7be21375-0635-48b2-8f4d-e3f74ea65359_isrForCrosstalkSources_855236_43.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7be21375-0635-48b2-8f4d-e3f74ea65359_isrForCrosstalkSources_855236_43.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 9d54ac1581d63535a2717fd4d18ef95e parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6ffe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d6619e73-8390-48e1-9d85-1b956b8791ca_isrForCrosstalkSources_855156_9.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d6619e73-8390-48e1-9d85-1b956b8791ca_isrForCrosstalkSources_855156_9.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 8cf38d8eb6b27e7d2334783584c4a259 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/29ee462d-96dd-442c-964e-a04035a1179d_isrForCrosstalkSources_855231_43.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/29ee462d-96dd-442c-964e-a04035a1179d_isrForCrosstalkSources_855231_43.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash f5adee9c465744757d4646239bbffa0e parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed6fc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9b20e6d0-9a91-4811-93bf-5dbce9bfd85b_isrForCrosstalkSources_855233_11.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9b20e6d0-9a91-4811-93bf-5dbce9bfd85b_isrForCrosstalkSources_855233_11.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash c574ea27aed944222518a8b424d89e5b parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/64e55775-f9fc-4c5b-85b9-76917a0c5619_isrForCrosstalkSources_855241_52.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/64e55775-f9fc-4c5b-85b9-76917a0c5619_isrForCrosstalkSources_855241_52.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 8e4b3bf5d2c0ecb5fcb3c90a6944b5ba parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d3736718-a427-4053-bd96-8b87e546c63a_isrForCrosstalkSources_855154_24.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d3736718-a427-4053-bd96-8b87e546c63a_isrForCrosstalkSources_855154_24.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash ea71b1af9d7342a117f8076653629348 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/220002d9-0a69-478c-a7d6-cb6fc3f1b4a3_isrForCrosstalkSources_855232_4.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/220002d9-0a69-478c-a7d6-cb6fc3f1b4a3_isrForCrosstalkSources_855232_4.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 1599178ed4a0b017dd663a696d6e84e0 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/74f67774-abb8-4d1b-9005-828fc172781e_isrForCrosstalkSources_855233_28.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/74f67774-abb8-4d1b-9005-828fc172781e_isrForCrosstalkSources_855233_28.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash f121b6b4da8d3ccec97aea7325e80fc8 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f42b2cd1-49c4-42ba-a106-cc3c302c3b06_isrForCrosstalkSources_855235_34.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f42b2cd1-49c4-42ba-a106-cc3c302c3b06_isrForCrosstalkSources_855235_34.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 7d896897f9f0c210073e3af979615046 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/73c4facd-6ef2-4a76-8cc7-8e9f3739b549_isrForCrosstalkSources_855156_5.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/73c4facd-6ef2-4a76-8cc7-8e9f3739b549_isrForCrosstalkSources_855156_5.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 547b281fa5b3162ed33fc7d0f5f8a011 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/dee42f89-5c85-4667-b60d-b75a56dddda6_isrForCrosstalkSources_855231_31.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/dee42f89-5c85-4667-b60d-b75a56dddda6_isrForCrosstalkSources_855231_31.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 4e977e5afe5b5cb85850ab1df30e978b parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/065fdd40-fae7-4f5f-b201-30a040e1e61e_isrForCrosstalkSources_855239_4.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/065fdd40-fae7-4f5f-b201-30a040e1e61e_isrForCrosstalkSources_855239_4.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 4f6938f1afb39ba401560962cd7d4127 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2e6d6b50-8330-4feb-826a-d5b8c95721b7_isrForCrosstalkSources_855155_45.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2e6d6b50-8330-4feb-826a-d5b8c95721b7_isrForCrosstalkSources_855155_45.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 7afd6b2bc22087f8bda14eca143f4157 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a11b9ecb-d230-46f5-a943-c829d0aa594a_isrForCrosstalkSources_855235_61.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a11b9ecb-d230-46f5-a943-c829d0aa594a_isrForCrosstalkSources_855235_61.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash ec705d257445ce67f0d10e2ebe593d92 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ea98ee2-d12c-44f8-98a9-9907814bb944_isrForCrosstalkSources_855236_26.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ea98ee2-d12c-44f8-98a9-9907814bb944_isrForCrosstalkSources_855236_26.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash bb7c60fe263759b59b2442dda4405135 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d9de2441-c958-4462-b50b-7ccb2f07cdd9_isrForCrosstalkSources_855240_9.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d9de2441-c958-4462-b50b-7ccb2f07cdd9_isrForCrosstalkSources_855240_9.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash d438a8c545a8a96de68aa4f0c34217d6 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c1d01ccf-78fc-4796-bfaf-d47b36244442_isrForCrosstalkSources_855232_12.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c1d01ccf-78fc-4796-bfaf-d47b36244442_isrForCrosstalkSources_855232_12.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 6ba8bc79d658f07fd036b0bca3c571d5 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e2de5018-3f4d-43d5-b56c-f9b764dd0a45_isrForCrosstalkSources_855238_16.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e2de5018-3f4d-43d5-b56c-f9b764dd0a45_isrForCrosstalkSources_855238_16.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash f9b3139130602fc20348e87f621d5bb9 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7b2306aa-9475-4a20-b2cc-20bd629cd807_isrForCrosstalkSources_855154_59.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7b2306aa-9475-4a20-b2cc-20bd629cd807_isrForCrosstalkSources_855154_59.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 8d644c69e3aa84f3a7af1f1467dc1b6d parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ac1015ae-26d3-41e5-941f-44f6f109d9de_isrForCrosstalkSources_855233_32.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ac1015ae-26d3-41e5-941f-44f6f109d9de_isrForCrosstalkSources_855233_32.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 6ee9051c4b3b66da31f9674cf5b1b1e1 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7751b8b5-0ffd-4847-adaf-a68de266f6dd_isrForCrosstalkSources_855232_24.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7751b8b5-0ffd-4847-adaf-a68de266f6dd_isrForCrosstalkSources_855232_24.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 076710caab03735490631e3ea402b4f1 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2dc61eaf-5fd7-4a8a-9133-f045ba88bcda_isrForCrosstalkSources_855231_20.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2dc61eaf-5fd7-4a8a-9133-f045ba88bcda_isrForCrosstalkSources_855231_20.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 3415538ecd5c08a40a169089d81b71cf parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d273d11b-fde1-45f4-b065-d17ab0274162_isrForCrosstalkSources_855240_18.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d273d11b-fde1-45f4-b065-d17ab0274162_isrForCrosstalkSources_855240_18.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 18630b15d2ef98eebae7beb7bc5cc953 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aa4f64b4-cae4-42be-b0fe-323a1238d31d_isrForCrosstalkSources_855233_45.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aa4f64b4-cae4-42be-b0fe-323a1238d31d_isrForCrosstalkSources_855233_45.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 81f5013044c566202ced590d1f2353d0 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b04370a6-6179-4349-81be-8413cc340ecc_isrForCrosstalkSources_855234_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b04370a6-6179-4349-81be-8413cc340ecc_isrForCrosstalkSources_855234_2.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash d6d8e18fd72b6de4fa722090a2e364bf parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:47 proc_lsst.multi:146 [INFO] found job 30716 in provider local proc_lsst.multi INFO: found job 30716 in provider local parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/85370ba3-60a8-47fc-819c-506a13ca2941_isrForCrosstalkSources_855156_45.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/85370ba3-60a8-47fc-819c-506a13ca2941_isrForCrosstalkSources_855156_45.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 3395db3c426319586736fa87d7491859 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528edd57ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c23c9313-ed9f-4157-ad66-4ca027c9c65c_isrForCrosstalkSources_855231_52.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c23c9313-ed9f-4157-ad66-4ca027c9c65c_isrForCrosstalkSources_855231_52.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 3e460606c90d20d82e9b32118c095c5b parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c655eac8-63d2-4ee7-80de-b2c46ef9b9dd_isrForCrosstalkSources_855239_55.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c655eac8-63d2-4ee7-80de-b2c46ef9b9dd_isrForCrosstalkSources_855239_55.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 9b64b53b64878290daaa899ab0c076cc parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b0a040ee-f483-4867-80d8-7e41710c7deb_isrForCrosstalkSources_855231_12.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b0a040ee-f483-4867-80d8-7e41710c7deb_isrForCrosstalkSources_855231_12.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash ee1abf3558974f519f5afd18af8416c1 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/55e84964-809a-480c-b850-adbd2234c0d9_isrForCrosstalkSources_855156_37.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/55e84964-809a-480c-b850-adbd2234c0d9_isrForCrosstalkSources_855156_37.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 150316ff2846ba8937349c73b57f3a66 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d54844df-f5a7-4052-b551-e045efbf968d_isrForCrosstalkSources_855233_59.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d54844df-f5a7-4052-b551-e045efbf968d_isrForCrosstalkSources_855233_59.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 8bdf53e3d5bfb2967007a1942a43aeda parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a2c11462-b1ef-4883-87e1-6125e2a07d14_isrForCrosstalkSources_855236_33.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a2c11462-b1ef-4883-87e1-6125e2a07d14_isrForCrosstalkSources_855236_33.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash e55f91bc91c116794448f18cea1425a5 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e9e42015-6c85-4c5a-aff6-e5381918364b_isrForCrosstalkSources_855154_29.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e9e42015-6c85-4c5a-aff6-e5381918364b_isrForCrosstalkSources_855154_29.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 60ca9df6fb3b0407b52cfb669fd3e9a5 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/58a64cad-cc71-4ad5-afc0-4272707e729b_isrForCrosstalkSources_855241_23.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/58a64cad-cc71-4ad5-afc0-4272707e729b_isrForCrosstalkSources_855241_23.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 97082a1295823c805e25f5b386dad097 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/adea0a8b-7bf1-41a2-8c3c-a8bd60aeac20_isrForCrosstalkSources_855231_15.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/adea0a8b-7bf1-41a2-8c3c-a8bd60aeac20_isrForCrosstalkSources_855231_15.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 67d2f1a81e324d4fe636a59ed453e1c9 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/db71e67c-f1b0-402a-8684-931665705fd6_isrForCrosstalkSources_855236_55.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/db71e67c-f1b0-402a-8684-931665705fd6_isrForCrosstalkSources_855236_55.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 70b91f2d44789480746dc7e5fb8003a9 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/22625fd5-104d-4d28-859e-032808520819_isrForCrosstalkSources_855238_27.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/22625fd5-104d-4d28-859e-032808520819_isrForCrosstalkSources_855238_27.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash bc6e13fc9c5f73b08e8b2e3eb2c144f6 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a07f1334-11b4-4db6-9ac8-c64c6903eb89_isrForCrosstalkSources_855240_40.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a07f1334-11b4-4db6-9ac8-c64c6903eb89_isrForCrosstalkSources_855240_40.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 55b0b22a8dae845440572f410343e835 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a94c35e3-9a2d-46c2-8bda-e89740db5329_isrForCrosstalkSources_855155_12.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a94c35e3-9a2d-46c2-8bda-e89740db5329_isrForCrosstalkSources_855155_12.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 73674f1c87221cf818e9bec0d14af619 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 152 parsl.dataflow.strategy DEBUG: Executor multi has 152 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 37 more blocks parsl.executors.status_handling INFO: Scaling out by 37 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-08 10:00:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-08 10:00:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 10:00:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ce136a19-970e-4281-84a2-c1713432452b_isrForCrosstalkSources_855237_28.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ce136a19-970e-4281-84a2-c1713432452b_isrForCrosstalkSources_855237_28.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 55b4fc07f7f426ba406b909223e42977 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/3958c07f-c0ca-4607-8434-5b564a9e3921_isrForCrosstalkSources_855239_37.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/3958c07f-c0ca-4607-8434-5b564a9e3921_isrForCrosstalkSources_855239_37.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash a3ecffad28c4a2135f6d59aa02aceccf parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d4e85563-77cb-4c44-8a3f-1c9356ea5717_isrForCrosstalkSources_855235_56.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d4e85563-77cb-4c44-8a3f-1c9356ea5717_isrForCrosstalkSources_855235_56.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash f7a9ab365a8ef27fa3d7523b2342d830 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/f7d491bf-0339-4014-9a17-49c7567ae08f_isrForCrosstalkSources_855231_1.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/f7d491bf-0339-4014-9a17-49c7567ae08f_isrForCrosstalkSources_855231_1.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 03ad803b9602203fefea17f33ca9e009 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a26e7107-edec-4056-81b7-1d5258990696_isrForCrosstalkSources_855235_1.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a26e7107-edec-4056-81b7-1d5258990696_isrForCrosstalkSources_855235_1.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 94010715d86d4c97ace3c6875a909a9e parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6536e09-2b95-461e-af91-a298bf623846_isrForCrosstalkSources_855240_24.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6536e09-2b95-461e-af91-a298bf623846_isrForCrosstalkSources_855240_24.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 6ccec982af078bd5ea9d0e653ae03be2 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/bbab30dc-5a2f-44bb-bda6-5b779882bb67_isrForCrosstalkSources_855242_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/bbab30dc-5a2f-44bb-bda6-5b779882bb67_isrForCrosstalkSources_855242_55.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 73a7294ac4d5042577472f9f73683ba6 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd84a1c4-d2d8-473a-b6a9-4ca70aac576a_isrForCrosstalkSources_855154_2.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd84a1c4-d2d8-473a-b6a9-4ca70aac576a_isrForCrosstalkSources_855154_2.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 6c5a0ed3922da97933410fa6ba776d09 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/98f3faea-a9ce-4c61-85e7-540d388e9e49_isrForCrosstalkSources_855242_62.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/98f3faea-a9ce-4c61-85e7-540d388e9e49_isrForCrosstalkSources_855242_62.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash f26cd605f327c34eea712003dd8427e4 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f825c3c4-7b9e-44de-beba-b0ff9a61ca49_isrForCrosstalkSources_855239_47.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f825c3c4-7b9e-44de-beba-b0ff9a61ca49_isrForCrosstalkSources_855239_47.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 15d3be00087820ef9d4b5c28e641e7b4 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/1c538154-72c3-4e15-8701-49f77d3100ca_isrForCrosstalkSources_855231_47.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/1c538154-72c3-4e15-8701-49f77d3100ca_isrForCrosstalkSources_855231_47.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash b8b818cdd7ae930041448bdfd46d9f28 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/0905e600-aee7-4216-a01a-e1cfefebd0c5_isrForCrosstalkSources_855240_14.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/0905e600-aee7-4216-a01a-e1cfefebd0c5_isrForCrosstalkSources_855240_14.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 6a0cf5dd7ffdbc2637b8202c3f94e7ec parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c11652e0-a895-4f64-81da-53f203406628_isrForCrosstalkSources_855231_26.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c11652e0-a895-4f64-81da-53f203406628_isrForCrosstalkSources_855231_26.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash f02c7a1f12a74e55bfec254d72e53f7d parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7d7e7f10-c107-4c05-bbbf-1eb58622a934_isrForCrosstalkSources_855231_48.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7d7e7f10-c107-4c05-bbbf-1eb58622a934_isrForCrosstalkSources_855231_48.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 877056c6ae000c0c81c895d76c417a82 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/79acced6-6c82-42b1-baa2-abba42fe8bd5_isrForCrosstalkSources_855232_46.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/79acced6-6c82-42b1-baa2-abba42fe8bd5_isrForCrosstalkSources_855232_46.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 50ca9f941630bb01bbac42a9655ae4d9 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/790d0944-db5e-430a-84c4-867787e9b911_isrForCrosstalkSources_855238_5.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/790d0944-db5e-430a-84c4-867787e9b911_isrForCrosstalkSources_855238_5.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 11c53ad79047ca497fa3f868cce8859a parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2050d7b8-740d-4cbe-83a7-1d38ef8b1389_isrForCrosstalkSources_855154_46.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2050d7b8-740d-4cbe-83a7-1d38ef8b1389_isrForCrosstalkSources_855154_46.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash fb72f6cb9ef010b06f29a626507217c9 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4fc4314d-fcf5-4294-aeaf-8b6d32359cd0_isrForCrosstalkSources_855242_13.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4fc4314d-fcf5-4294-aeaf-8b6d32359cd0_isrForCrosstalkSources_855242_13.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 096c4c4cbfd75f44f7918ec8628efa2c parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c45e14ad-0d91-4f31-a8d0-618a19d40fcb_isrForCrosstalkSources_855237_10.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c45e14ad-0d91-4f31-a8d0-618a19d40fcb_isrForCrosstalkSources_855237_10.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 08502c22e14fafa8c18c3fa9aca1edcd parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/444a57c6-2981-4098-9417-7a3d19e84cba_isrForCrosstalkSources_855242_26.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/444a57c6-2981-4098-9417-7a3d19e84cba_isrForCrosstalkSources_855242_26.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 669fb647c2c09c03c56015e5dd937ffc parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6bd0f52e-96ad-494b-940a-365c8b45071a_isrForCrosstalkSources_855156_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6bd0f52e-96ad-494b-940a-365c8b45071a_isrForCrosstalkSources_855156_2.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 43ac83fe8abb091219d7f2dfb3d30b21 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/43704e7b-f22d-4706-b7ee-08bcd42a514e_isrForCrosstalkSources_855235_10.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/43704e7b-f22d-4706-b7ee-08bcd42a514e_isrForCrosstalkSources_855235_10.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 02a2c45447585b12d11f4d6c824c021a parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/527efd63-f35a-48ee-bb27-05289e1d7a0b_isrForCrosstalkSources_855231_24.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/527efd63-f35a-48ee-bb27-05289e1d7a0b_isrForCrosstalkSources_855231_24.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash d8ceb7ab0f7edb96bac3296f03bbbe6f parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/80f169a8-cc4e-46aa-90de-5219a06b2422_isrForCrosstalkSources_855241_9.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/80f169a8-cc4e-46aa-90de-5219a06b2422_isrForCrosstalkSources_855241_9.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 09f5a1d846eb4a136495afd5e6cbbcbd parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/37795145-9588-4c00-a0bb-127b4d5ee98a_isrForCrosstalkSources_855234_61.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/37795145-9588-4c00-a0bb-127b4d5ee98a_isrForCrosstalkSources_855234_61.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 793f6cfec19e1035524fb598bf8e008a parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5ea22d61-ffc0-4c5d-9585-1314433e9e9b_isrForCrosstalkSources_855233_24.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5ea22d61-ffc0-4c5d-9585-1314433e9e9b_isrForCrosstalkSources_855233_24.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 59f7bfe25464b6270a6a93fa646d9c2a parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a773ce8b-706e-4e13-93a9-4156df5e08da_isrForCrosstalkSources_855238_20.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a773ce8b-706e-4e13-93a9-4156df5e08da_isrForCrosstalkSources_855238_20.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 3a61283117804e1a756c03f2bc704f2e parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/9664a0b8-53f4-422a-a37e-41343a0d7828_isrForCrosstalkSources_855156_58.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/9664a0b8-53f4-422a-a37e-41343a0d7828_isrForCrosstalkSources_855156_58.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 2b13d596ce7b9455825f401f3bd54323 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/edba68ad-b074-4e9a-a713-86bd87cb1494_isrForCrosstalkSources_855156_46.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/edba68ad-b074-4e9a-a713-86bd87cb1494_isrForCrosstalkSources_855156_46.stderr 2024-03-08 10:00:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914600 proc_lsst.multi INFO: [multi] job_id 16914600 2024-03-08 10:00:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914600 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914600 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16914600 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-08 10:00:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-08 10:00:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 62dd01c670b71ad109a010404932e529 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/5c7ba540-d26d-4d44-83d1-412cf7a7876a_isrForCrosstalkSources_855232_41.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/5c7ba540-d26d-4d44-83d1-412cf7a7876a_isrForCrosstalkSources_855232_41.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 0a825e295af3a322394756be9c3efb41 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a5a98d09-e153-41b9-92c5-33108c6473ae_isrForCrosstalkSources_855238_41.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a5a98d09-e153-41b9-92c5-33108c6473ae_isrForCrosstalkSources_855238_41.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 78874388f6c3d1413eb8547e4b4f0df2 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/17787604-2e8a-44b4-a209-a82653f3d3fe_isrForCrosstalkSources_855235_44.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/17787604-2e8a-44b4-a209-a82653f3d3fe_isrForCrosstalkSources_855235_44.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 88a5d77dcc5b6f7f380e9366b56c8c87 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f548239-fd73-443d-bddd-d1f31898d7b9_isrForCrosstalkSources_855241_19.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f548239-fd73-443d-bddd-d1f31898d7b9_isrForCrosstalkSources_855241_19.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 1a79e7ffa8ea9e0cc3cabe64a463f35c parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc549441-aec6-4782-b9c8-3e40f098b558_isrForCrosstalkSources_855234_19.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc549441-aec6-4782-b9c8-3e40f098b558_isrForCrosstalkSources_855234_19.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 416a43b93c7ac0f8905541fe6438c4aa parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d71d90b7-fd6d-416f-9cc1-f83b26f755fb_isrForCrosstalkSources_855237_59.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d71d90b7-fd6d-416f-9cc1-f83b26f755fb_isrForCrosstalkSources_855237_59.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash f8a3a6cecc3bc78f9cb1d2aff6183913 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93626210-e536-46f8-a83a-d389ca3121c4_isrForCrosstalkSources_855155_15.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93626210-e536-46f8-a83a-d389ca3121c4_isrForCrosstalkSources_855155_15.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash b3ce7986cb61839834e8056313885f55 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c32a9d6f-7a44-445c-8fc8-2478e923c69a_isrForCrosstalkSources_855237_11.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c32a9d6f-7a44-445c-8fc8-2478e923c69a_isrForCrosstalkSources_855237_11.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash c4d42456f420f6e4966f64c94e677fb1 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3025eb4c-61e3-4cf4-abf9-8cb73593aa01_isrForCrosstalkSources_855237_36.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3025eb4c-61e3-4cf4-abf9-8cb73593aa01_isrForCrosstalkSources_855237_36.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 045fb612fa2cbdebeb45f96fcd179657 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6b572aec-1b2b-4f31-a09f-31547d21b80b_isrForCrosstalkSources_855233_48.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6b572aec-1b2b-4f31-a09f-31547d21b80b_isrForCrosstalkSources_855233_48.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 9643169e52c4074b4ac101f84ee5ea95 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/dedbc313-eafc-4e6b-bcf9-8439a915f620_isrForCrosstalkSources_855235_41.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/dedbc313-eafc-4e6b-bcf9-8439a915f620_isrForCrosstalkSources_855235_41.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 5bbb06471ad93d0e9085cb9dd14f4c65 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1a06d35a-6dbb-43ca-bf64-963b348b0b5e_isrForCrosstalkSources_855238_28.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1a06d35a-6dbb-43ca-bf64-963b348b0b5e_isrForCrosstalkSources_855238_28.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 299682af4a3850a4bcddd36c4b12e5d9 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc9a2663-7c9d-4f8e-961e-5522236a8507_isrForCrosstalkSources_855235_58.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc9a2663-7c9d-4f8e-961e-5522236a8507_isrForCrosstalkSources_855235_58.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash be898cd7820eac490f076cc3d4587290 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e673ce2d-afa9-427d-9848-15666a5d1d96_isrForCrosstalkSources_855154_30.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e673ce2d-afa9-427d-9848-15666a5d1d96_isrForCrosstalkSources_855154_30.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 425f38f53a6a0f78e191c39685fed37e parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8124743b-a73b-4abc-93cc-bb9cca132bb9_isrForCrosstalkSources_855239_58.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8124743b-a73b-4abc-93cc-bb9cca132bb9_isrForCrosstalkSources_855239_58.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 2d24ab51aec63e80b8613034eea7b02d parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6f0acc6-5ae5-4d29-b597-333beac4d038_isrForCrosstalkSources_855232_53.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6f0acc6-5ae5-4d29-b597-333beac4d038_isrForCrosstalkSources_855232_53.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 5dbd4a3c0be9dc82e7b440ed47b906cd parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/81f09af6-83c3-4c97-9c77-8b7d474cb4e9_isrForCrosstalkSources_855236_60.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/81f09af6-83c3-4c97-9c77-8b7d474cb4e9_isrForCrosstalkSources_855236_60.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash e77a93d4f402892a80b938b3cf0bceb0 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bb71d1c-9d86-49ba-9d82-06c4191d7be0_isrForCrosstalkSources_855231_57.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bb71d1c-9d86-49ba-9d82-06c4191d7be0_isrForCrosstalkSources_855231_57.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 105b534103c29a35b6659f79b6ece258 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/422ad764-8d2f-4ab5-b7de-aaf5ccfca40a_isrForCrosstalkSources_855156_44.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/422ad764-8d2f-4ab5-b7de-aaf5ccfca40a_isrForCrosstalkSources_855156_44.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 9fd5a46521f0805f933afead45181da6 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ef28b32e-57b5-477b-b709-f47ac2cd05ad_isrForCrosstalkSources_855155_47.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ef28b32e-57b5-477b-b709-f47ac2cd05ad_isrForCrosstalkSources_855155_47.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash b4dcf1da2025a8065bdfe2944d3a10b9 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/c0cffadf-64b2-41b9-938b-a7f9140ca3ce_isrForCrosstalkSources_855234_40.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/c0cffadf-64b2-41b9-938b-a7f9140ca3ce_isrForCrosstalkSources_855234_40.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash f410c4e69fc2abc66847fd07665dbf54 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/835a3ed5-153c-423e-8626-ba402dbc37b7_isrForCrosstalkSources_855155_34.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/835a3ed5-153c-423e-8626-ba402dbc37b7_isrForCrosstalkSources_855155_34.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 99d6328e05dfa3a77ec057e78826c997 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a3c5444c-cd44-4b6a-8a93-e48122973e55_isrForCrosstalkSources_855234_6.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a3c5444c-cd44-4b6a-8a93-e48122973e55_isrForCrosstalkSources_855234_6.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 8cf38565c463bf93ac26a6df5f15d269 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/be912ab7-fc8a-4822-9cba-ae8e9bed788e_isrForCrosstalkSources_855232_36.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/be912ab7-fc8a-4822-9cba-ae8e9bed788e_isrForCrosstalkSources_855232_36.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 01652aa8c097711eb2c94d1d8c00bdc9 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0dd0705f-b5e9-4083-9801-85038425b5b3_isrForCrosstalkSources_855239_59.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0dd0705f-b5e9-4083-9801-85038425b5b3_isrForCrosstalkSources_855239_59.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash a1ddb5b27867f65149a10d06103fb718 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/687110ba-6c2c-4a7a-a705-c8fb4bdf05e6_isrForCrosstalkSources_855237_41.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/687110ba-6c2c-4a7a-a705-c8fb4bdf05e6_isrForCrosstalkSources_855237_41.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash f63f6f2e2c1e8c81ea7490c51f0f47b8 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/376e5909-6ae4-405b-a632-eb6194f066e8_isrForCrosstalkSources_855240_7.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/376e5909-6ae4-405b-a632-eb6194f066e8_isrForCrosstalkSources_855240_7.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash d61aac6f4e95af8cf75fc6b2a3af4d25 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab5336e6-df99-42dd-ad60-8896b734dac2_isrForCrosstalkSources_855235_6.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab5336e6-df99-42dd-ad60-8896b734dac2_isrForCrosstalkSources_855235_6.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 185407fcd36a7d0838ec20cb849e2658 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a401edc5-a2c0-4bd4-a729-b24abf7ffb0e_isrForCrosstalkSources_855237_3.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a401edc5-a2c0-4bd4-a729-b24abf7ffb0e_isrForCrosstalkSources_855237_3.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash f4fb0abf65720d5bd3f17a6738018c23 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/5c35f1bb-8614-4025-acf6-f5b51624489e_isrForCrosstalkSources_855240_54.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/5c35f1bb-8614-4025-acf6-f5b51624489e_isrForCrosstalkSources_855240_54.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash d0a79cc115c60a05151e4b5ff253f3d6 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/49eb0497-d028-4699-9480-ccb504661d07_isrForCrosstalkSources_855237_39.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/49eb0497-d028-4699-9480-ccb504661d07_isrForCrosstalkSources_855237_39.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 0b1f6effbfb66dfcb1ffe9fa198cf4bf parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914601 proc_lsst.multi INFO: [multi] job_id 16914601 2024-03-08 10:00:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:00:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914601 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914601 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16914601 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-08 10:00:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/9cd98147-48d7-45c5-be6f-448cad0c4b94_isrForCrosstalkSources_855242_45.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/9cd98147-48d7-45c5-be6f-448cad0c4b94_isrForCrosstalkSources_855242_45.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash b69aa071408765008e1b68c3a2087bc2 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b9e58d2a-1521-47f6-9e16-a19fd0dfca06_isrForCrosstalkSources_855234_9.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b9e58d2a-1521-47f6-9e16-a19fd0dfca06_isrForCrosstalkSources_855234_9.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 77ab53ced259861f724b5c195b394c5c parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/38c40461-1175-4a52-89fe-0f51c4003f4f_isrForCrosstalkSources_855232_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/38c40461-1175-4a52-89fe-0f51c4003f4f_isrForCrosstalkSources_855232_17.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 9b49fc10d068f87bef13ecc403fcef63 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1a9b05b0-8a09-49e6-a23a-f95c5f3772b7_isrForCrosstalkSources_855240_36.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1a9b05b0-8a09-49e6-a23a-f95c5f3772b7_isrForCrosstalkSources_855240_36.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 2f06167dd53228374da07fa31fd57334 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/41602420-cdb7-40e6-9ce9-2bf06fc2caae_isrForCrosstalkSources_855155_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/41602420-cdb7-40e6-9ce9-2bf06fc2caae_isrForCrosstalkSources_855155_2.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash e1b18ff71ff82fced2e91b7fc6af5403 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/25c25ab0-4877-4a1e-9eb3-4a8bce3b88f3_isrForCrosstalkSources_855234_3.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/25c25ab0-4877-4a1e-9eb3-4a8bce3b88f3_isrForCrosstalkSources_855234_3.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash a15f10e27fc82fcf8a3444b0a088c0f8 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d9e87ab0-c8df-4c90-8a85-5c7044c6b538_isrForCrosstalkSources_855154_47.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d9e87ab0-c8df-4c90-8a85-5c7044c6b538_isrForCrosstalkSources_855154_47.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 4f3bbeb12cee53e8f99d9434e24d4225 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/42d77067-b130-4faf-bf77-1cc39db43c68_isrForCrosstalkSources_855234_58.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/42d77067-b130-4faf-bf77-1cc39db43c68_isrForCrosstalkSources_855234_58.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash bb69cabc4f22bb0747deb844fbe8c7bc parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d18f768e-efb2-4552-bef8-bc944f45572a_isrForCrosstalkSources_855240_59.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d18f768e-efb2-4552-bef8-bc944f45572a_isrForCrosstalkSources_855240_59.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 8ad8e76b603aba9b0a7211d4e5b3ec84 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/19277ea6-6202-4c31-a507-98b472f58865_isrForCrosstalkSources_855235_49.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/19277ea6-6202-4c31-a507-98b472f58865_isrForCrosstalkSources_855235_49.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 8e18137a0803867780b86f3b082f9337 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9de9a0d6-5ae9-48a9-b7fd-042687fdbc86_isrForCrosstalkSources_855154_15.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9de9a0d6-5ae9-48a9-b7fd-042687fdbc86_isrForCrosstalkSources_855154_15.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 00bdc0c6c05511711abce53af846e654 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/50f44755-afa3-42c6-bc1f-e7e99ef2f3b2_isrForCrosstalkSources_855154_12.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/50f44755-afa3-42c6-bc1f-e7e99ef2f3b2_isrForCrosstalkSources_855154_12.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash fed5912cd2db33f31ca95a2cedee5973 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/61150b6f-73a6-47ad-bcdf-27cee42e8478_isrForCrosstalkSources_855236_56.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/61150b6f-73a6-47ad-bcdf-27cee42e8478_isrForCrosstalkSources_855236_56.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash e65362246129ef05859c174337577fdb parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e5383b5a-201b-4b77-bab6-e62ddf825e08_isrForCrosstalkSources_855242_6.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e5383b5a-201b-4b77-bab6-e62ddf825e08_isrForCrosstalkSources_855242_6.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 2b728e87f6a9142f6bab6eb497c66e5d parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/51cbb6fd-3b64-4f71-ad86-15738e87ce5f_isrForCrosstalkSources_855236_38.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/51cbb6fd-3b64-4f71-ad86-15738e87ce5f_isrForCrosstalkSources_855236_38.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 2529091968c3bbf6b926b9744472d3bb parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d4c6a4db-b391-4dcf-936c-48ac8f29bb53_isrForCrosstalkSources_855232_20.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d4c6a4db-b391-4dcf-936c-48ac8f29bb53_isrForCrosstalkSources_855232_20.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 8253a82b276d006864189d981ff2b904 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c884bb95-5939-45ec-ab31-6bcb63a9b26e_isrForCrosstalkSources_855154_25.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c884bb95-5939-45ec-ab31-6bcb63a9b26e_isrForCrosstalkSources_855154_25.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash fd586c948805168f71dc77337e101161 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/29e3e857-4783-46e9-b97a-a6bc0adacaaf_isrForCrosstalkSources_855241_45.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/29e3e857-4783-46e9-b97a-a6bc0adacaaf_isrForCrosstalkSources_855241_45.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 461c827e0e4094602b161fd5d57f9416 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc3aac43-8214-4b0b-b06e-fb98499e096d_isrForCrosstalkSources_855235_27.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc3aac43-8214-4b0b-b06e-fb98499e096d_isrForCrosstalkSources_855235_27.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 29f2fcdab54bcdb14196eceafe79ec05 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d2c834ff-a195-4e96-91db-1fd556639d20_isrForCrosstalkSources_855231_21.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d2c834ff-a195-4e96-91db-1fd556639d20_isrForCrosstalkSources_855231_21.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 89a65b471500a8751a39d207f7ca2c8e parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8e53925-1a4b-4436-9d4e-3b5265c3402d_isrForCrosstalkSources_855155_10.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8e53925-1a4b-4436-9d4e-3b5265c3402d_isrForCrosstalkSources_855155_10.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 547c6646c062a7de719b9cdd65be63f0 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/bd83980e-71e4-472c-b510-a9302ac8487b_isrForCrosstalkSources_855234_17.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/bd83980e-71e4-472c-b510-a9302ac8487b_isrForCrosstalkSources_855234_17.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 5af40706c84ba07ccd5ad76ea768919a parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/0d766dc9-a6d5-451d-ab5c-0e4f5ad56121_isrForCrosstalkSources_855236_10.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/0d766dc9-a6d5-451d-ab5c-0e4f5ad56121_isrForCrosstalkSources_855236_10.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash b06e060c1a9efa62e064b54e45cfcabb parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e0a2eb93-503c-46f1-8cfc-4d0ada6daa58_isrForCrosstalkSources_855241_6.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e0a2eb93-503c-46f1-8cfc-4d0ada6daa58_isrForCrosstalkSources_855241_6.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 63debddf92497e30883b8c0b3215a3fd parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/646ae36a-1052-4a0a-92b1-e60f74c129b0_isrForCrosstalkSources_855234_22.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/646ae36a-1052-4a0a-92b1-e60f74c129b0_isrForCrosstalkSources_855234_22.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 2105aeb5d5a98475686228fbe28d7983 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/79f87437-5365-45e5-9c07-1cef594f42d2_isrForCrosstalkSources_855156_35.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/79f87437-5365-45e5-9c07-1cef594f42d2_isrForCrosstalkSources_855156_35.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash b6bc078ebae12a8410099a4e31f4c430 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a816879e-4e24-431c-ad61-d76c76e34910_isrForCrosstalkSources_855240_16.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a816879e-4e24-431c-ad61-d76c76e34910_isrForCrosstalkSources_855240_16.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash bc76388d9063ca1eb280028e706f0cf0 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cee0d7d2-3981-45d3-96d2-45d1820bab50_isrForCrosstalkSources_855241_10.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cee0d7d2-3981-45d3-96d2-45d1820bab50_isrForCrosstalkSources_855241_10.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash e82a57e92151524fb55c9bf4ec72ce7d parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/234e4b0d-afef-4578-ad92-085896aede53_isrForCrosstalkSources_855238_56.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/234e4b0d-afef-4578-ad92-085896aede53_isrForCrosstalkSources_855238_56.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash d234b0b2487b6f5d79beebdbb2bf9bb2 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914602 proc_lsst.multi INFO: [multi] job_id 16914602 2024-03-08 10:00:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:00:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914602 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914602 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16914602 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-08 10:00:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1648f305-d415-41f0-9075-537e073ae294_isrForCrosstalkSources_855239_28.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1648f305-d415-41f0-9075-537e073ae294_isrForCrosstalkSources_855239_28.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash ff3ace6aca7cd7c7416610e3d9b11b35 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/26d09a68-8a4c-49e7-83e8-9058c3679adb_isrForCrosstalkSources_855155_50.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/26d09a68-8a4c-49e7-83e8-9058c3679adb_isrForCrosstalkSources_855155_50.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash efa2cfaae6d8e6d346c084ad3764dcfe parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0bb776ef-2aea-4fb9-80c3-af8cc8653456_isrForCrosstalkSources_855237_44.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0bb776ef-2aea-4fb9-80c3-af8cc8653456_isrForCrosstalkSources_855237_44.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash f772f9a3705a7283c78b4075024d4f19 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c37b71c6-bc3f-4bb7-8f99-c4512136c437_isrForCrosstalkSources_855231_60.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c37b71c6-bc3f-4bb7-8f99-c4512136c437_isrForCrosstalkSources_855231_60.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 0495e649b5955769516180fb61b1e6b1 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/36ba3895-4bb1-4e24-bb2a-a3430259d8f1_isrForCrosstalkSources_855240_30.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/36ba3895-4bb1-4e24-bb2a-a3430259d8f1_isrForCrosstalkSources_855240_30.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash c7a88a8794936441324ef33f5a231852 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/80db1a87-9015-4a94-b298-79424e6f7562_isrForCrosstalkSources_855240_37.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/80db1a87-9015-4a94-b298-79424e6f7562_isrForCrosstalkSources_855240_37.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash c6f5e73bc4824fcb3091db3e71d68c1e parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f686054b-8ca1-46e0-bf91-8f022a9fc2e3_isrForCrosstalkSources_855155_16.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f686054b-8ca1-46e0-bf91-8f022a9fc2e3_isrForCrosstalkSources_855155_16.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash beb7a134c54ef91c347cf953cbaa27e4 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a50bee3f-f8fc-4f2c-b9af-a0fa72dbe09a_isrForCrosstalkSources_855239_19.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a50bee3f-f8fc-4f2c-b9af-a0fa72dbe09a_isrForCrosstalkSources_855239_19.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash a8a224f8861b87eeefb1293a77eae159 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a40ee240-0e29-4e41-aa73-b0ad31ec57f3_isrForCrosstalkSources_855242_12.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a40ee240-0e29-4e41-aa73-b0ad31ec57f3_isrForCrosstalkSources_855242_12.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash a99c5e65640fd75ddbf6cdca661031c9 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d9134325-c22d-4382-b87f-2f3f998bcddf_isrForCrosstalkSources_855155_59.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d9134325-c22d-4382-b87f-2f3f998bcddf_isrForCrosstalkSources_855155_59.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 81509b18190e5789d4cdd2861aa96dc6 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/5ff50ad6-3924-4f1b-90c1-3d84af93e13b_isrForCrosstalkSources_855231_33.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/5ff50ad6-3924-4f1b-90c1-3d84af93e13b_isrForCrosstalkSources_855231_33.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash f963bcf99f918dcb3aa136e98c3d5500 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1aded576-8dd2-4d1a-aaad-18d23df55154_isrForCrosstalkSources_855239_34.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1aded576-8dd2-4d1a-aaad-18d23df55154_isrForCrosstalkSources_855239_34.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash af95b01b02faa397c85cfdf8151b7c8a parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/88c6a692-b32c-451b-bf69-a1683d153131_isrForCrosstalkSources_855240_62.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/88c6a692-b32c-451b-bf69-a1683d153131_isrForCrosstalkSources_855240_62.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 642a7d29b687fc6e427847aa3d345a51 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/49c2d75b-9c8d-4c02-bcce-a8bfc42afa08_isrForCrosstalkSources_855239_20.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/49c2d75b-9c8d-4c02-bcce-a8bfc42afa08_isrForCrosstalkSources_855239_20.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash d142d6ebd1e79aca6526c909d64fbc8c parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/11ddc760-2714-45f7-9e1f-e0314e373a75_isrForCrosstalkSources_855235_35.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/11ddc760-2714-45f7-9e1f-e0314e373a75_isrForCrosstalkSources_855235_35.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash f24308cb8ce555197671fe7faaf6136b parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6f730530-4331-4c65-9467-7ea06264cef0_isrForCrosstalkSources_855155_48.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6f730530-4331-4c65-9467-7ea06264cef0_isrForCrosstalkSources_855155_48.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 3ffa3d37176ef6e3d94dc69988520104 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/383b4d4c-9a92-496c-bdd6-58b32de906c6_isrForCrosstalkSources_855231_38.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/383b4d4c-9a92-496c-bdd6-58b32de906c6_isrForCrosstalkSources_855231_38.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 363132f882ee8109852c3785851a5c14 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c914cda3-b866-4238-ac48-6b9b6ac5b043_isrForCrosstalkSources_855240_53.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c914cda3-b866-4238-ac48-6b9b6ac5b043_isrForCrosstalkSources_855240_53.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 87bbfe2dec8f496a76e724a218c53769 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b04f56e4-2a13-4e4e-b035-d1c0b43b7f9a_isrForCrosstalkSources_855154_54.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b04f56e4-2a13-4e4e-b035-d1c0b43b7f9a_isrForCrosstalkSources_855154_54.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 80f7d2610a11b8d11312def965dc9360 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/875b242f-f0a6-491c-b7b8-9f2d632a7933_isrForCrosstalkSources_855156_20.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/875b242f-f0a6-491c-b7b8-9f2d632a7933_isrForCrosstalkSources_855156_20.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash ffa5e68f08998c71d8008185357329c2 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e7e8965a-60af-4e4e-8d17-2d9d0847978b_isrForCrosstalkSources_855242_48.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e7e8965a-60af-4e4e-8d17-2d9d0847978b_isrForCrosstalkSources_855242_48.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 8a0cb3c5c18a01cfc70380b48e445341 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fa75006e-d635-4709-82b9-e833770e19b8_isrForCrosstalkSources_855235_45.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fa75006e-d635-4709-82b9-e833770e19b8_isrForCrosstalkSources_855235_45.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash d93aba22532957b160d7f9e879ac6dd4 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/38f23734-95fa-4565-93d0-b58addd55708_isrForCrosstalkSources_855235_25.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/38f23734-95fa-4565-93d0-b58addd55708_isrForCrosstalkSources_855235_25.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash b4eaf448498a7da5e90b807c30f71f20 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3f8fb23b-8fa6-4b95-aa0c-120fda607d85_isrForCrosstalkSources_855156_32.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3f8fb23b-8fa6-4b95-aa0c-120fda607d85_isrForCrosstalkSources_855156_32.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 89d07624b62f034239ee6c9453a3625c parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/eafafeb5-2076-405d-8daf-c368d655cf1e_isrForCrosstalkSources_855237_23.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/eafafeb5-2076-405d-8daf-c368d655cf1e_isrForCrosstalkSources_855237_23.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 41f8f073ec58a3573b6bb12ff7017f94 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f9b2da64-974e-490e-9b03-99d7afba3994_isrForCrosstalkSources_855234_59.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f9b2da64-974e-490e-9b03-99d7afba3994_isrForCrosstalkSources_855234_59.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 5f742d3e398e9830d4ac381a5ff737c5 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c11c7160-a5ff-4b43-a11f-fcf6d024bffc_isrForCrosstalkSources_855156_10.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c11c7160-a5ff-4b43-a11f-fcf6d024bffc_isrForCrosstalkSources_855156_10.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 25328a7ee75250c336df1293d1985656 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/e99b6079-1ccf-4600-a72d-1e812416c20e_isrForCrosstalkSources_855236_29.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/e99b6079-1ccf-4600-a72d-1e812416c20e_isrForCrosstalkSources_855236_29.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 06533f6c90fe397190d40f674ce29bab parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2fe6392a-63b3-466f-bec7-3409b9885ec7_isrForCrosstalkSources_855238_43.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2fe6392a-63b3-466f-bec7-3409b9885ec7_isrForCrosstalkSources_855238_43.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash ecf59bac2e23d530ba6ea9b46c1f0349 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6905d954-4a5a-4e42-8c43-c658a2c74823_isrForCrosstalkSources_855231_36.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6905d954-4a5a-4e42-8c43-c658a2c74823_isrForCrosstalkSources_855231_36.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 518570ecc95c1d4f30efbea969132012 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/51ec8ca5-fbf9-4156-a209-1c52564c2790_isrForCrosstalkSources_855242_33.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/51ec8ca5-fbf9-4156-a209-1c52564c2790_isrForCrosstalkSources_855242_33.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 5e44567cec1d754c05712601815b99aa parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/cbac35c0-7518-41e6-8bf5-82600b0822f4_isrForCrosstalkSources_855154_17.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/cbac35c0-7518-41e6-8bf5-82600b0822f4_isrForCrosstalkSources_855154_17.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 6139cf7eae9571a770d15a7b0f7b8ff8 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7f5eb7dc-7200-43cd-9bbe-72954b2e2bd0_isrForCrosstalkSources_855231_35.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7f5eb7dc-7200-43cd-9bbe-72954b2e2bd0_isrForCrosstalkSources_855231_35.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash d581f70f552d1c057a7e7b81b28d9757 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/1c9605d5-cad0-497e-8abb-85d2c260ae64_isrForCrosstalkSources_855235_15.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/1c9605d5-cad0-497e-8abb-85d2c260ae64_isrForCrosstalkSources_855235_15.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies 2024-03-08 10:00:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914603 proc_lsst.multi INFO: [multi] job_id 16914603 2024-03-08 10:00:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:00:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914603 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914603 parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 4de99f4234a00b4769e41b6a317c61ad parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16914603 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-08 10:00:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/3c7edd30-8946-4264-bd8a-c5bda121af2b_isrForCrosstalkSources_855154_62.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/3c7edd30-8946-4264-bd8a-c5bda121af2b_isrForCrosstalkSources_855154_62.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 80141809fb27e8f71b987f95c7abfdc0 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03866d3e-3030-4285-8da8-8ef92b000e3a_isrForCrosstalkSources_855238_46.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03866d3e-3030-4285-8da8-8ef92b000e3a_isrForCrosstalkSources_855238_46.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 260fc24ed270a4e4b498406b27525cac parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd95eb13-4336-4112-b8a3-c4cf6822019b_isrForCrosstalkSources_855154_37.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd95eb13-4336-4112-b8a3-c4cf6822019b_isrForCrosstalkSources_855154_37.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 6e00c411eeb16c2ff7715ecb6b4a7ec5 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/68a3f9eb-8155-494b-8c2c-ba896b654ff3_isrForCrosstalkSources_855156_53.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/68a3f9eb-8155-494b-8c2c-ba896b654ff3_isrForCrosstalkSources_855156_53.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash f33153922803e28060765abfd8e6c587 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/59079c17-83eb-4022-997a-b9dcd7436943_isrForCrosstalkSources_855239_15.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/59079c17-83eb-4022-997a-b9dcd7436943_isrForCrosstalkSources_855239_15.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 9c1810b783c66225fe70da2065c7783c parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/4cf480c1-6157-47ca-85df-b9d7bc22940d_isrForCrosstalkSources_855232_35.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/4cf480c1-6157-47ca-85df-b9d7bc22940d_isrForCrosstalkSources_855232_35.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash f7f1a32a3236f9d66793280064e8eb49 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/17978ef6-5e21-48ef-ba48-c934f7c012f2_isrForCrosstalkSources_855242_31.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/17978ef6-5e21-48ef-ba48-c934f7c012f2_isrForCrosstalkSources_855242_31.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 4f04d24aa319f40b02897e4f7e2609e1 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0f82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2de19804-fb88-476e-bb82-a1ad67804f53_isrForCrosstalkSources_855239_13.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2de19804-fb88-476e-bb82-a1ad67804f53_isrForCrosstalkSources_855239_13.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 6b7439e558ae5cf382acceea5c71f201 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9f0c673f-76c2-434e-ba17-ba987e2be7e1_isrForCrosstalkSources_855232_15.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9f0c673f-76c2-434e-ba17-ba987e2be7e1_isrForCrosstalkSources_855232_15.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash c71173a9cd6c7054f25bc69fbe0d35a3 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ab01297-239d-4a85-ba31-567d4ac64649_isrForCrosstalkSources_855234_21.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ab01297-239d-4a85-ba31-567d4ac64649_isrForCrosstalkSources_855234_21.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 3c7bb515103ff38a35c8ada5eb80f9db parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/26bacb6c-ad5f-4e52-a21e-1d0ac8b91fe3_isrForCrosstalkSources_855236_35.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/26bacb6c-ad5f-4e52-a21e-1d0ac8b91fe3_isrForCrosstalkSources_855236_35.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 370a0813b622554d2a3a46c7170105f9 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0faf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/68507f1d-ea1d-4cf8-8e73-a49652750aa8_isrForCrosstalkSources_855240_39.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/68507f1d-ea1d-4cf8-8e73-a49652750aa8_isrForCrosstalkSources_855240_39.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 69cb09e63c26b876532ad360adb4ceaf parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bee72959-208d-4df2-a827-aa63266e4ac3_isrForCrosstalkSources_855240_45.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bee72959-208d-4df2-a827-aa63266e4ac3_isrForCrosstalkSources_855240_45.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash d26125b08a5473c767da15cfd818547c parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0faca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/596cd7a3-e5dc-4a6a-8435-0cfeba205f1d_isrForCrosstalkSources_855240_25.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/596cd7a3-e5dc-4a6a-8435-0cfeba205f1d_isrForCrosstalkSources_855240_25.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 132c719817784594c7e13838221756ef parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/cea92e5d-3c49-4f8a-9f2c-e79138ccae01_isrForCrosstalkSources_855235_11.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/cea92e5d-3c49-4f8a-9f2c-e79138ccae01_isrForCrosstalkSources_855235_11.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 97467e854461a4d5018ca93b9b81ace5 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8e8b2326-323b-46da-86b8-edf2e6a61b95_isrForCrosstalkSources_855154_50.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8e8b2326-323b-46da-86b8-edf2e6a61b95_isrForCrosstalkSources_855154_50.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 7771fcb142c88b4c8241bf4d150a9378 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0faac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3b0e667d-5cc1-407a-bc64-f4b712f1ceac_isrForCrosstalkSources_855234_48.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3b0e667d-5cc1-407a-bc64-f4b712f1ceac_isrForCrosstalkSources_855234_48.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 16be16eedce12398a5e0412bc1b24e59 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0faa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8f6c9629-f2c7-4655-8710-6b90ef1317c3_isrForCrosstalkSources_855237_34.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8f6c9629-f2c7-4655-8710-6b90ef1317c3_isrForCrosstalkSources_855237_34.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 9225ed30d7ecb647650d53f20ebf5f8d parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/98edcfc3-8180-41ca-91e8-7677426db419_isrForCrosstalkSources_855236_3.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/98edcfc3-8180-41ca-91e8-7677426db419_isrForCrosstalkSources_855236_3.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 406d28becc66542b570b30e345dc0700 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d2753ee3-e266-4c5b-9fd1-d5ec95666787_isrForCrosstalkSources_855238_36.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d2753ee3-e266-4c5b-9fd1-d5ec95666787_isrForCrosstalkSources_855238_36.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 814477059d0da3025b79c694dbc42903 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/b3c48d53-e9b6-4fdd-89cc-ece47604a381_isrForCrosstalkSources_855237_19.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/b3c48d53-e9b6-4fdd-89cc-ece47604a381_isrForCrosstalkSources_855237_19.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 71166ebbac9d0e48adc86719781e1813 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914604 proc_lsst.multi INFO: [multi] job_id 16914604 2024-03-08 10:00:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914604 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914604 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16914604 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-08 10:00:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9a852e51-7613-4c14-80df-5d96863521d1_isrForCrosstalkSources_855240_15.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9a852e51-7613-4c14-80df-5d96863521d1_isrForCrosstalkSources_855240_15.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 8c4758cfc985c2db99f263cd7b3b5db4 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528fdd377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a5d9a0f-9b0d-4fb4-9e10-2767eb1624d5_isrForCrosstalkSources_855237_52.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a5d9a0f-9b0d-4fb4-9e10-2767eb1624d5_isrForCrosstalkSources_855237_52.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash dd7363b1c92e08d0bb178da82132dbdc parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0f102005-8229-4a85-be2a-c94099fbc670_isrForCrosstalkSources_855234_20.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0f102005-8229-4a85-be2a-c94099fbc670_isrForCrosstalkSources_855234_20.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash da32e6179cfc00655426d543bf3551ab parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c33ac37f-b7c9-42ab-a846-75ebf84bed13_isrForCrosstalkSources_855237_46.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c33ac37f-b7c9-42ab-a846-75ebf84bed13_isrForCrosstalkSources_855237_46.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash e9cdb658b14cad275c3b6d2d00415e87 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/27ff2633-cd87-4e6b-8878-32c66e400f3d_isrForCrosstalkSources_855236_30.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/27ff2633-cd87-4e6b-8878-32c66e400f3d_isrForCrosstalkSources_855236_30.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 41f4e0365b95045338e40e8dd6f8c26d parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ef62c870-5539-420b-83c0-901649aeea0d_isrForCrosstalkSources_855233_17.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ef62c870-5539-420b-83c0-901649aeea0d_isrForCrosstalkSources_855233_17.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 1f8462e43d16e4e97003d1d9e81bb21f parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f8f956fe-d5c9-4c5a-b3b7-908214249f93_isrForCrosstalkSources_855155_3.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f8f956fe-d5c9-4c5a-b3b7-908214249f93_isrForCrosstalkSources_855155_3.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash d86a4d5f6b6cc068b1b26e14ab80cb1f parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a94dbcd0-a4a3-4f65-a9f5-d571693caf51_isrForCrosstalkSources_855231_28.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a94dbcd0-a4a3-4f65-a9f5-d571693caf51_isrForCrosstalkSources_855231_28.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 847d535dab16c377162f1576b40e16df parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e59026cd-2627-467e-a74e-0dffcb9a4e7b_isrForCrosstalkSources_855231_22.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e59026cd-2627-467e-a74e-0dffcb9a4e7b_isrForCrosstalkSources_855231_22.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 762e3f8db23c732c246c2829a3b9ceea parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8fb053b7-aabc-4e3e-9f12-79ae8f3f5091_isrForCrosstalkSources_855232_3.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8fb053b7-aabc-4e3e-9f12-79ae8f3f5091_isrForCrosstalkSources_855232_3.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 64d347d25170db8ec352d0de1b4fb67c parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/69efa9fa-6565-4ce2-b4b5-ed980c584b78_isrForCrosstalkSources_855236_39.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/69efa9fa-6565-4ce2-b4b5-ed980c584b78_isrForCrosstalkSources_855236_39.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 6374b4151d853dc279db3a39e4cbec0f parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/fb2e9e81-1bd6-4879-94c2-5df987f2bea6_isrForCrosstalkSources_855232_51.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/fb2e9e81-1bd6-4879-94c2-5df987f2bea6_isrForCrosstalkSources_855232_51.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 5d848dcd4a953c4008f7692b66191176 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/09dc70b1-7250-4563-a782-e9d095d66f95_isrForCrosstalkSources_855154_4.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/09dc70b1-7250-4563-a782-e9d095d66f95_isrForCrosstalkSources_855154_4.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash cc80a1aa88bd983780c5f615cafaea65 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/19dac0f7-28c5-4d1f-a09d-e20a7bc651b3_isrForCrosstalkSources_855236_54.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/19dac0f7-28c5-4d1f-a09d-e20a7bc651b3_isrForCrosstalkSources_855236_54.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 83bdac19513c1f2e14e2572dd64c1405 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8a404fd5-849e-4715-ab75-d5c93c99f1e9_isrForCrosstalkSources_855234_36.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8a404fd5-849e-4715-ab75-d5c93c99f1e9_isrForCrosstalkSources_855234_36.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash b4566ae375caeaf65cf8a62309e377b6 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/86417a52-f1e2-45e9-8885-467e9cc9b34c_isrForCrosstalkSources_855237_45.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/86417a52-f1e2-45e9-8885-467e9cc9b34c_isrForCrosstalkSources_855237_45.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 052127d772da8612d873214431c9629e parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3531398-a054-489a-8856-c977138875ab_isrForCrosstalkSources_855235_17.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3531398-a054-489a-8856-c977138875ab_isrForCrosstalkSources_855235_17.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 17132ba6233463b7c11d02a8ceed8634 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f19614c0-4405-4a62-b103-52ad4a14af6e_isrForCrosstalkSources_855242_4.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f19614c0-4405-4a62-b103-52ad4a14af6e_isrForCrosstalkSources_855242_4.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 6dce3665f715af7279b2abc773d5542c parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7a3db405-09f7-4c64-a88a-d3b05dff977f_isrForCrosstalkSources_855155_39.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7a3db405-09f7-4c64-a88a-d3b05dff977f_isrForCrosstalkSources_855155_39.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 30e65d1fa98d42ca89b865951e9fa1bc parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f8641062-b30e-4cdf-a07d-029e46f62389_isrForCrosstalkSources_855239_62.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f8641062-b30e-4cdf-a07d-029e46f62389_isrForCrosstalkSources_855239_62.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash f1eef1c46938c20ad8dac5a1d0543e5c parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b949b79-d557-4e9a-8f9e-3ed7122ef5e2_isrForCrosstalkSources_855241_44.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b949b79-d557-4e9a-8f9e-3ed7122ef5e2_isrForCrosstalkSources_855241_44.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 966587334c5d6adfa08d4da736b10141 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ef6d5ad8-08ed-4ef1-b2c7-2e47a762a2ff_isrForCrosstalkSources_855239_29.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ef6d5ad8-08ed-4ef1-b2c7-2e47a762a2ff_isrForCrosstalkSources_855239_29.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash f9a2a1d86ec5630bdf066402a69bec48 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8c830a3c-97c7-46e8-abeb-2d76748ca361_isrForCrosstalkSources_855236_28.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8c830a3c-97c7-46e8-abeb-2d76748ca361_isrForCrosstalkSources_855236_28.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 4370716f5aae8beaf72a422ec2423bd3 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b8f5b2b9-3f32-4747-bc10-c1618002e85e_isrForCrosstalkSources_855232_27.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b8f5b2b9-3f32-4747-bc10-c1618002e85e_isrForCrosstalkSources_855232_27.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash f83b96e38f56842c66bbd5ed13bb2137 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f01ec94f-5116-4a54-b272-4c0eb1e1a7e9_isrForCrosstalkSources_855156_50.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f01ec94f-5116-4a54-b272-4c0eb1e1a7e9_isrForCrosstalkSources_855156_50.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 5af65a707bf71c86a7d01fe80a97222c parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/1f8728e6-6c21-48e4-96af-fd07b32f339c_isrForCrosstalkSources_855156_6.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/1f8728e6-6c21-48e4-96af-fd07b32f339c_isrForCrosstalkSources_855156_6.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash d6e6dc9ae01212d0057c92f4e88fc068 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c32d7ea4-1956-4210-b978-e0d6a6049a15_isrForCrosstalkSources_855240_42.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c32d7ea4-1956-4210-b978-e0d6a6049a15_isrForCrosstalkSources_855240_42.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 15cd9b34a9c6de1fa0ba643c22837039 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ccba2a76-8795-4604-8212-229dd94f927a_isrForCrosstalkSources_855242_8.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ccba2a76-8795-4604-8212-229dd94f927a_isrForCrosstalkSources_855242_8.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 796865f5ca71e9363bced37c11b7f4bb parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6d2012c1-5288-47a2-a0d7-dd09979555a3_isrForCrosstalkSources_855154_19.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6d2012c1-5288-47a2-a0d7-dd09979555a3_isrForCrosstalkSources_855154_19.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 9357ea507dd03a40f2cd003297a111fb parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/403e657a-0f44-4035-b957-8492863a7117_isrForCrosstalkSources_855156_28.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/403e657a-0f44-4035-b957-8492863a7117_isrForCrosstalkSources_855156_28.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 9fdc02e7a50e52460c09f576f859c608 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/daa83ac4-9e20-491d-a7ef-c457f949c9b2_isrForCrosstalkSources_855236_23.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/daa83ac4-9e20-491d-a7ef-c457f949c9b2_isrForCrosstalkSources_855236_23.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 8ed7c9c3e977fd0d2debb018281bd7ef parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/66373349-6031-481f-ab72-7894f5a109c0_isrForCrosstalkSources_855233_7.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/66373349-6031-481f-ab72-7894f5a109c0_isrForCrosstalkSources_855233_7.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash d3321cf3c060d7e5377da9edc2498ed9 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0a7468ed-b63d-4955-afa6-d985a59b3026_isrForCrosstalkSources_855242_9.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0a7468ed-b63d-4955-afa6-d985a59b3026_isrForCrosstalkSources_855242_9.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 691f1267ab716d7d9a4a996a907b0231 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6ed331a-fc0f-4efa-ae6e-6192d7d50ac2_isrForCrosstalkSources_855232_30.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6ed331a-fc0f-4efa-ae6e-6192d7d50ac2_isrForCrosstalkSources_855232_30.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash b344e2b33550584d5f1b592c78eeb7eb parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1db546ce-5808-4184-9c12-cfa5b81196ac_isrForCrosstalkSources_855155_30.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1db546ce-5808-4184-9c12-cfa5b81196ac_isrForCrosstalkSources_855155_30.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash a243f25ffb5bf7c62a5181f2f3213bd8 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/55767d2b-33ee-44c7-b24e-d11487d92484_isrForCrosstalkSources_855234_25.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/55767d2b-33ee-44c7-b24e-d11487d92484_isrForCrosstalkSources_855234_25.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash cdfdde20bf1c475347735a41c5786f22 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a8a0a5bf-83c2-47e9-baa0-1f53ad591362_isrForCrosstalkSources_855237_31.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a8a0a5bf-83c2-47e9-baa0-1f53ad591362_isrForCrosstalkSources_855237_31.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash a32ec297ba35bd52a8af4adfca39e546 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/323c6cff-ca98-48cc-97c4-263144c4b466_isrForCrosstalkSources_855238_40.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/323c6cff-ca98-48cc-97c4-263144c4b466_isrForCrosstalkSources_855238_40.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash a1398ff9a129c8cfab348c5f7d9337d2 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4c7f9581-27e9-4ec7-b534-8e72d263ac8c_isrForCrosstalkSources_855156_42.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4c7f9581-27e9-4ec7-b534-8e72d263ac8c_isrForCrosstalkSources_855156_42.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash be18a58d831bad01e31a046ea1530548 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/82bcd5f1-e3f5-43d5-9bb6-00454056dc11_isrForCrosstalkSources_855241_31.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/82bcd5f1-e3f5-43d5-9bb6-00454056dc11_isrForCrosstalkSources_855241_31.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 05bf4c004c8a6449e0d84100f071fae2 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/caa2711f-4821-4f08-b8ed-0f013fa59262_isrForCrosstalkSources_855232_39.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/caa2711f-4821-4f08-b8ed-0f013fa59262_isrForCrosstalkSources_855232_39.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash 0b7cf3b33e6f69037a2471fc729bd78f parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7facc6ce-81ba-483e-bdc6-60aeb010a8cb_isrForCrosstalkSources_855241_1.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7facc6ce-81ba-483e-bdc6-60aeb010a8cb_isrForCrosstalkSources_855241_1.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 5f4995d530a7009e17f613c9d4a3b61f parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/83fbbf08-3e43-45e4-a019-c8f4e9a4f39e_isrForCrosstalkSources_855234_7.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/83fbbf08-3e43-45e4-a019-c8f4e9a4f39e_isrForCrosstalkSources_855234_7.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash a4e22e1bce626c87504bfd1b46728d28 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5919188b-184e-4705-ab75-9dda660af2a1_isrForCrosstalkSources_855242_18.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5919188b-184e-4705-ab75-9dda660af2a1_isrForCrosstalkSources_855242_18.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash e45e9de9ad7616d3ab3ca91089737580 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/85446f4d-8427-406e-a0ad-81c92fe06f50_isrForCrosstalkSources_855236_2.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/85446f4d-8427-406e-a0ad-81c92fe06f50_isrForCrosstalkSources_855236_2.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 467ff2b0e32f7f734820f6055fc0c7e0 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56ede5e0-66e9-4ba0-a89c-ff09e3867dcc_isrForCrosstalkSources_855237_54.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56ede5e0-66e9-4ba0-a89c-ff09e3867dcc_isrForCrosstalkSources_855237_54.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash d2206ad3ecf85a9f524fe080520697f2 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914605 proc_lsst.multi INFO: [multi] job_id 16914605 2024-03-08 10:00:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 10:00:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914605 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914605 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16914605 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-08 10:00:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-08 10:00:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 10:00:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a317ca89-f8d7-404d-99b7-678a02323853_isrForCrosstalkSources_855235_14.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a317ca89-f8d7-404d-99b7-678a02323853_isrForCrosstalkSources_855235_14.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash b872999d4357917f64ded46acfb076b5 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d58930d-78d4-4518-8f45-11536c8b421a_isrForCrosstalkSources_855154_18.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d58930d-78d4-4518-8f45-11536c8b421a_isrForCrosstalkSources_855154_18.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 7e9b90fd7c91fdac71cf2bf4632bad9e parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6b1b8f70-8db1-4216-bdfb-f83b67705a1c_isrForCrosstalkSources_855231_39.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6b1b8f70-8db1-4216-bdfb-f83b67705a1c_isrForCrosstalkSources_855231_39.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 9bdc55f5d1434da9ffb4853cdfb97d3b parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aeb658cd-3ce3-4075-91f1-a7555ac06717_isrForCrosstalkSources_855242_7.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aeb658cd-3ce3-4075-91f1-a7555ac06717_isrForCrosstalkSources_855242_7.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 4dab14cf87ede0e6aa396e1fdc493f04 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1c2933c1-f47a-4745-a56f-e390009ca0bc_isrForCrosstalkSources_855232_26.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1c2933c1-f47a-4745-a56f-e390009ca0bc_isrForCrosstalkSources_855232_26.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 719fd30ecb394d234cd416cd45fc3fe0 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3d4441f7-aa51-4bbd-9266-284899365cf5_isrForCrosstalkSources_855231_51.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3d4441f7-aa51-4bbd-9266-284899365cf5_isrForCrosstalkSources_855231_51.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 94e750e5888ad5a1dd49ec3fafb5cdc5 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c7b61f91-942b-47cb-9b6b-272ab2f65464_isrForCrosstalkSources_855156_21.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c7b61f91-942b-47cb-9b6b-272ab2f65464_isrForCrosstalkSources_855156_21.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash eff3b19ff0d9d1aefc71cec2d5f2be4a parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6665bbe7-a806-44ec-ac35-b8593ebe1376_isrForCrosstalkSources_855241_35.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6665bbe7-a806-44ec-ac35-b8593ebe1376_isrForCrosstalkSources_855241_35.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash a59088a8ebaee588294cfb2ab6706d67 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68008abe-4a16-4a89-8bdd-3ac54438d68f_isrForCrosstalkSources_855238_38.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68008abe-4a16-4a89-8bdd-3ac54438d68f_isrForCrosstalkSources_855238_38.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 88f5e94c541bca6cf4b059571c796794 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e8e70a47-bb8c-4b0a-af95-18be3dfc7a67_isrForCrosstalkSources_855235_43.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e8e70a47-bb8c-4b0a-af95-18be3dfc7a67_isrForCrosstalkSources_855235_43.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash bbb53c65ee646af28c61e1a8aab8c557 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/086d9e08-b07c-435d-990c-11471bf1c008_isrForCrosstalkSources_855154_20.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/086d9e08-b07c-435d-990c-11471bf1c008_isrForCrosstalkSources_855154_20.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 3a4b1233da79b0e2295e367ee0e8285b parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9e02a26-b670-4989-a289-635ff9fece81_isrForCrosstalkSources_855156_7.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9e02a26-b670-4989-a289-635ff9fece81_isrForCrosstalkSources_855156_7.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 0238ce9bfe0f38ba7d6347075f837e15 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7eda13b7-b0c7-408b-87b8-c7d8007c667e_isrForCrosstalkSources_855234_29.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7eda13b7-b0c7-408b-87b8-c7d8007c667e_isrForCrosstalkSources_855234_29.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 278f8b0b36bc76543db74c1f9345eef1 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6c1a158c-2a8b-45f4-a47c-3adc3a675b3b_isrForCrosstalkSources_855156_41.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6c1a158c-2a8b-45f4-a47c-3adc3a675b3b_isrForCrosstalkSources_855156_41.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash ead730dfcb294165e4be35dc20723e53 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/90db3adb-5900-4d1e-ba10-2c80e15601e8_isrForCrosstalkSources_855233_12.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/90db3adb-5900-4d1e-ba10-2c80e15601e8_isrForCrosstalkSources_855233_12.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash b8991a9d3a56e71bea11f2c00c312efa parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5240718c-40b2-4337-86f6-97cd49b4e934_isrForCrosstalkSources_855235_16.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5240718c-40b2-4337-86f6-97cd49b4e934_isrForCrosstalkSources_855235_16.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash d971c90ee94a43ef5ff71f0033f56598 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a6ce6229-6cb8-4407-97bc-fdec02fb9a9f_isrForCrosstalkSources_855232_44.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a6ce6229-6cb8-4407-97bc-fdec02fb9a9f_isrForCrosstalkSources_855232_44.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 4bd9437d00724aac25478132f3274dc6 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ff32a981-1609-4ab4-a5e9-6237d1493d22_isrForCrosstalkSources_855235_5.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ff32a981-1609-4ab4-a5e9-6237d1493d22_isrForCrosstalkSources_855235_5.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash eaf4fac547333b3f5b464e995e1d6825 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/50bc7779-1d15-4441-ba70-e7b605c5d0e6_isrForCrosstalkSources_855231_45.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/50bc7779-1d15-4441-ba70-e7b605c5d0e6_isrForCrosstalkSources_855231_45.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 10fd002830f6ff2f2821cf855de22e8b parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f1033c6f-8546-46b4-bdb9-2edab5da9d28_isrForCrosstalkSources_855156_60.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f1033c6f-8546-46b4-bdb9-2edab5da9d28_isrForCrosstalkSources_855156_60.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 52a24f2de64068fb74d3a44545b301dd parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a88de9e-c240-4bc8-bee8-e112e6765cbc_isrForCrosstalkSources_855235_31.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a88de9e-c240-4bc8-bee8-e112e6765cbc_isrForCrosstalkSources_855235_31.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash e728b1f334049fd4910c1abe13fca5d8 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/64c691ea-70dc-4dae-b2e8-356f82f7dd7f_isrForCrosstalkSources_855232_47.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/64c691ea-70dc-4dae-b2e8-356f82f7dd7f_isrForCrosstalkSources_855232_47.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 4f33b855f8ed61185c7f3664ecc56df4 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5207160d-3701-4096-ae21-00818136e64b_isrForCrosstalkSources_855233_44.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5207160d-3701-4096-ae21-00818136e64b_isrForCrosstalkSources_855233_44.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash ec9f02fbbdcfe8d04278d3d6305c98b2 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9831555a-bf28-4bc0-a478-cc235dd07cea_isrForCrosstalkSources_855239_35.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9831555a-bf28-4bc0-a478-cc235dd07cea_isrForCrosstalkSources_855239_35.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 0c9dd2161ff5fa0fd4f6c126d85770cf parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6ac4c800-b0f9-4d7d-a85d-21510240e70d_isrForCrosstalkSources_855242_17.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6ac4c800-b0f9-4d7d-a85d-21510240e70d_isrForCrosstalkSources_855242_17.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 961e25c0df547d36325883bdd5d9e826 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d88cf86-881d-4fe1-8e9b-f11c1b7f4c06_isrForCrosstalkSources_855237_24.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d88cf86-881d-4fe1-8e9b-f11c1b7f4c06_isrForCrosstalkSources_855237_24.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash cffe8bcf30c81554ff954ae2f627648a parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2bc2ef5a-3f87-4529-a2b4-bdd41252b1c2_isrForCrosstalkSources_855240_31.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2bc2ef5a-3f87-4529-a2b4-bdd41252b1c2_isrForCrosstalkSources_855240_31.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash ca53068b33dda7054cc23f29da9a2284 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/54e57280-85d5-471e-aa13-5f430f284210_isrForCrosstalkSources_855155_1.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/54e57280-85d5-471e-aa13-5f430f284210_isrForCrosstalkSources_855155_1.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash ab89f79e0c2ec4e200a0d85b16aa62b3 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/49db40f7-86d4-46e8-a913-7fa37dc323c4_isrForCrosstalkSources_855234_1.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/49db40f7-86d4-46e8-a913-7fa37dc323c4_isrForCrosstalkSources_855234_1.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash d96c98f798ada4eae7f9992eb8d2ec8b parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e6a22cc3-beaa-406a-a9ec-e9c217c4479d_isrForCrosstalkSources_855232_11.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e6a22cc3-beaa-406a-a9ec-e9c217c4479d_isrForCrosstalkSources_855232_11.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 1b2501b9046c51c4060321baca1de315 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8ce4a5af-ff8d-4032-8b64-289cd14ed975_isrForCrosstalkSources_855154_48.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8ce4a5af-ff8d-4032-8b64-289cd14ed975_isrForCrosstalkSources_855154_48.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 6dcbfde080634256afb8f7d8904c8a78 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914606 proc_lsst.multi INFO: [multi] job_id 16914606 2024-03-08 10:00:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 10:00:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914606 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914606 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16914606 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-08 10:00:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-08 10:00:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 10:00:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/25cc7374-c3a0-4a72-b951-92bf3f69f792_isrForCrosstalkSources_855238_52.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/25cc7374-c3a0-4a72-b951-92bf3f69f792_isrForCrosstalkSources_855238_52.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 219654c64c3eb23d6d299cadb06aea8a parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ffe8c9f2-f654-457e-865e-980d127596fe_isrForCrosstalkSources_855236_62.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ffe8c9f2-f654-457e-865e-980d127596fe_isrForCrosstalkSources_855236_62.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 17ea4d7cce89a0afd4b46d5a8dccbcce parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20c76607-16b8-4bab-a8fc-eaa1bec8d5d0_isrForCrosstalkSources_855156_1.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20c76607-16b8-4bab-a8fc-eaa1bec8d5d0_isrForCrosstalkSources_855156_1.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 0f1f5d602e99d7a2fcde84020fe2600f parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fa7c5d43-af3e-4f51-8e09-caedc403b86a_isrForCrosstalkSources_855239_33.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fa7c5d43-af3e-4f51-8e09-caedc403b86a_isrForCrosstalkSources_855239_33.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash defe19145a2bad124eee141b1c8ef45a parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b46fe0ab-5a51-4620-9700-b4c9ff89f207_isrForCrosstalkSources_855241_50.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b46fe0ab-5a51-4620-9700-b4c9ff89f207_isrForCrosstalkSources_855241_50.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash d620abff0cab20d23f0ccc32089cafaf parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/816fe5b0-4e10-4dfc-8452-3b5b05f8fb34_isrForCrosstalkSources_855237_56.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/816fe5b0-4e10-4dfc-8452-3b5b05f8fb34_isrForCrosstalkSources_855237_56.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash ecf5ef5d08b068f077c3c974e8569365 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c095cdff-2d34-411a-801e-53dc01a02236_isrForCrosstalkSources_855239_23.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c095cdff-2d34-411a-801e-53dc01a02236_isrForCrosstalkSources_855239_23.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 9c19277551bb609454495f9631e8178d parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/92d1ecb2-78ef-4260-ab51-de961802480e_isrForCrosstalkSources_855155_8.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/92d1ecb2-78ef-4260-ab51-de961802480e_isrForCrosstalkSources_855155_8.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash bf4aad9c6de4b3e5adef6f1bc875ff0c parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2411d2e9-e4cf-4293-a98a-8b486b4bee9f_isrForCrosstalkSources_855231_14.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2411d2e9-e4cf-4293-a98a-8b486b4bee9f_isrForCrosstalkSources_855231_14.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 45bfb7e9aabd44a8b0a576145712c08d parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c114dfd3-52eb-488e-880c-488a8ecf4ea4_isrForCrosstalkSources_855155_25.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c114dfd3-52eb-488e-880c-488a8ecf4ea4_isrForCrosstalkSources_855155_25.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash b29b02151e509f7672629bcf97883551 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/422ee4ea-fbf0-4123-8c57-4d8b4b3b7a1e_isrForCrosstalkSources_855233_26.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/422ee4ea-fbf0-4123-8c57-4d8b4b3b7a1e_isrForCrosstalkSources_855233_26.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 3ed1f6ad26ed71cbafca504479d59c72 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b7d1277e-67b5-4673-bcce-5f7808676f7f_isrForCrosstalkSources_855232_6.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b7d1277e-67b5-4673-bcce-5f7808676f7f_isrForCrosstalkSources_855232_6.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash b83241f79dd0c6ae19aa6d13e1e16083 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/434dba8d-47ef-4d5e-887e-9f099ca70848_isrForCrosstalkSources_855236_50.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/434dba8d-47ef-4d5e-887e-9f099ca70848_isrForCrosstalkSources_855236_50.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 21f547f43cf5f81f2b6b1f0351ab1ed1 parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/54e5f70f-7e67-47b3-b726-cbfde1b4e59b_isrForCrosstalkSources_855156_18.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/54e5f70f-7e67-47b3-b726-cbfde1b4e59b_isrForCrosstalkSources_855156_18.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 180a7eb145ff04e42b1ce6b7a9995712 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4a89aca6-9180-42bf-abaf-e63f56584d40_isrForCrosstalkSources_855231_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4a89aca6-9180-42bf-abaf-e63f56584d40_isrForCrosstalkSources_855231_2.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash ff64234e76ab1e9acb65b245b9b642de parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3412af94-a3de-47ff-a982-ecc0925b2b46_isrForCrosstalkSources_855237_20.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3412af94-a3de-47ff-a982-ecc0925b2b46_isrForCrosstalkSources_855237_20.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash ba2583d81d38c169b5ce3f8cc7fd6ea3 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0d27690c-25d3-4b41-be05-493453547140_isrForCrosstalkSources_855239_57.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0d27690c-25d3-4b41-be05-493453547140_isrForCrosstalkSources_855239_57.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash b9e4770e3a8f3908ecfd01812d9c8ebe parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68d930e7-e624-4249-89b7-ba9cb33c60eb_isrForCrosstalkSources_855238_4.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68d930e7-e624-4249-89b7-ba9cb33c60eb_isrForCrosstalkSources_855238_4.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash b2d5e13c9b6055d0a20c102137f16c70 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ebf80c40-0c48-417f-ad37-73b2838edf1c_isrForCrosstalkSources_855236_34.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ebf80c40-0c48-417f-ad37-73b2838edf1c_isrForCrosstalkSources_855236_34.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash da090dc54e2786a2fe0f010d45905424 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/02c18129-53e5-4ac5-b664-3acf00977d27_isrForCrosstalkSources_855155_51.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/02c18129-53e5-4ac5-b664-3acf00977d27_isrForCrosstalkSources_855155_51.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 357c4f13adbe2d48e71dad884268008e parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/92cbe666-a7b1-4e84-8132-1cc4f7b54f0e_isrForCrosstalkSources_855154_51.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/92cbe666-a7b1-4e84-8132-1cc4f7b54f0e_isrForCrosstalkSources_855154_51.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash a5dee9a5bd9fd860827b04be2096647a parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b96b4bb4-eba5-443b-85b1-d1dc296d014c_isrForCrosstalkSources_855242_21.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b96b4bb4-eba5-443b-85b1-d1dc296d014c_isrForCrosstalkSources_855242_21.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 0b3b50bbb5e92427436ca2753d8effb6 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f7e62045-4e19-42fd-92b0-ddb8fa801b2d_isrForCrosstalkSources_855235_18.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f7e62045-4e19-42fd-92b0-ddb8fa801b2d_isrForCrosstalkSources_855235_18.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 752834fc5bcf0d1ab7fbad64dbe65965 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/69e55859-64d0-41f4-a005-e7f28fa2839a_isrForCrosstalkSources_855232_10.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/69e55859-64d0-41f4-a005-e7f28fa2839a_isrForCrosstalkSources_855232_10.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 09331432f0312b98939036d94adb885d parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a4d8063-b2af-42a9-a3f6-c6ae99e555f6_isrForCrosstalkSources_855156_36.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a4d8063-b2af-42a9-a3f6-c6ae99e555f6_isrForCrosstalkSources_855156_36.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash a4f9c3ad9b5fc72db8697d73e354a322 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914607 proc_lsst.multi INFO: [multi] job_id 16914607 2024-03-08 10:00:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 10:00:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914607 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914607 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16914607 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-08 10:00:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-08 10:00:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 10:00:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fc06db14-d9d3-4eab-b56e-55ab67e9a5bf_isrForCrosstalkSources_855238_22.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fc06db14-d9d3-4eab-b56e-55ab67e9a5bf_isrForCrosstalkSources_855238_22.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 696d53ea1e07e656ca2ec97265a6dd91 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ce1cb304-0bbf-4d8a-b779-07b18fc9a6cf_isrForCrosstalkSources_855242_49.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ce1cb304-0bbf-4d8a-b779-07b18fc9a6cf_isrForCrosstalkSources_855242_49.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 06e4e0703ea04c871846aede743c3c5a parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/65b601b8-b34a-492a-a6e3-6437edb1a9f2_isrForCrosstalkSources_855238_44.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/65b601b8-b34a-492a-a6e3-6437edb1a9f2_isrForCrosstalkSources_855238_44.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 61b3399a023ef0a606d04e29306428d3 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/9b850888-d5c4-428f-aa2b-3d564316bebd_isrForCrosstalkSources_855235_13.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/9b850888-d5c4-428f-aa2b-3d564316bebd_isrForCrosstalkSources_855235_13.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 7dc664ac691d2f81e8a633aa9b0131cd parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/b357fff7-dc45-47d7-b56f-aaa37f3c847b_isrForCrosstalkSources_855156_16.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/b357fff7-dc45-47d7-b56f-aaa37f3c847b_isrForCrosstalkSources_855156_16.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash f6e02efecc58a395026440fb44781d13 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec0fb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8a9b12ac-5d31-43db-a997-e255a0f92467_isrForCrosstalkSources_855239_22.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8a9b12ac-5d31-43db-a997-e255a0f92467_isrForCrosstalkSources_855239_22.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash dd1297007a360d0329f08ba17fa7ca21 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/194da3ff-4ef9-4461-a9a5-0cc11a3bf855_isrForCrosstalkSources_855156_59.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/194da3ff-4ef9-4461-a9a5-0cc11a3bf855_isrForCrosstalkSources_855156_59.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash af7e20026fd7ab27581cf6063e246e0e parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3eac354a-f8f9-4c89-b456-14c292123719_isrForCrosstalkSources_855241_18.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3eac354a-f8f9-4c89-b456-14c292123719_isrForCrosstalkSources_855241_18.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 92b75649da4479a5ecac14f1df3058e4 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/689479e6-a188-4f34-8e84-04a640f481ba_isrForCrosstalkSources_855242_29.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/689479e6-a188-4f34-8e84-04a640f481ba_isrForCrosstalkSources_855242_29.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 932f084f07816f813144fd257d686aaf parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2eb0b5c3-6121-4081-a4f5-e4a55b8e9763_isrForCrosstalkSources_855155_58.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2eb0b5c3-6121-4081-a4f5-e4a55b8e9763_isrForCrosstalkSources_855155_58.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 0639fc79681e9dea23aace6fa60b1490 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2cbeb5d-23c1-4118-a0f8-3c89e100b28f_isrForCrosstalkSources_855239_27.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2cbeb5d-23c1-4118-a0f8-3c89e100b28f_isrForCrosstalkSources_855239_27.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash f762b365dcf089f343f2b1f8c51610dc parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/419118bf-abb1-4063-8542-b16acd4b9850_isrForCrosstalkSources_855232_55.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/419118bf-abb1-4063-8542-b16acd4b9850_isrForCrosstalkSources_855232_55.stderr parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 3b9d27ed7a617205b3175c715a1dad82 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0a72e735-1aa9-49fc-a5e1-a0695e1d1398_isrForCrosstalkSources_855156_23.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0a72e735-1aa9-49fc-a5e1-a0695e1d1398_isrForCrosstalkSources_855156_23.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 422f55135b5ff8064b08a21937d848ac parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7d8717fe-14b7-477f-b687-0854aa024f64_isrForCrosstalkSources_855154_6.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7d8717fe-14b7-477f-b687-0854aa024f64_isrForCrosstalkSources_855154_6.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 47be653f2ac4ae4d8bdce502615674a2 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3c5fc83b-4319-426b-b821-bedbb3140815_isrForCrosstalkSources_855234_15.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3c5fc83b-4319-426b-b821-bedbb3140815_isrForCrosstalkSources_855234_15.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash f3fc48a77a24331c343e77b503983d29 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5eb1823a-abf8-42df-857e-c41507920c70_isrForCrosstalkSources_855236_11.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5eb1823a-abf8-42df-857e-c41507920c70_isrForCrosstalkSources_855236_11.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 10ff623f8fcb714a1bc2dc199e410652 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b285ad8b-e571-4926-8f54-bcb5b238ea27_isrForCrosstalkSources_855239_61.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b285ad8b-e571-4926-8f54-bcb5b238ea27_isrForCrosstalkSources_855239_61.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 73bac566590cf2d12aaf50284f11270f parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/34b908df-9b82-4b92-b9f5-1c94ebf59ef3_isrForCrosstalkSources_855234_14.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/34b908df-9b82-4b92-b9f5-1c94ebf59ef3_isrForCrosstalkSources_855234_14.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash b74378fdc4476500ffc280f11914217d parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f7b24122-194f-4c62-953f-28068027c044_isrForCrosstalkSources_855234_55.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f7b24122-194f-4c62-953f-28068027c044_isrForCrosstalkSources_855234_55.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 22026ff049e1aa045fdec112304fd019 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a552ecd-32e9-4f85-be05-00ca5077475a_isrForCrosstalkSources_855156_8.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a552ecd-32e9-4f85-be05-00ca5077475a_isrForCrosstalkSources_855156_8.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 968e56c73b58cd5388239752495587ad parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/168437e9-b751-4cc3-b4fb-190b4ac6f579_isrForCrosstalkSources_855154_33.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/168437e9-b751-4cc3-b4fb-190b4ac6f579_isrForCrosstalkSources_855154_33.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 55aa3941f94f1153d9a2a101f0314bcd parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d7c253b8-b959-433f-8be9-c82a119c216e_isrForCrosstalkSources_855240_20.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d7c253b8-b959-433f-8be9-c82a119c216e_isrForCrosstalkSources_855240_20.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 7b52dc9849b340f444d1ea508a586d05 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/8a048b0a-19f0-42ef-b065-eb26e9997c00_isrForCrosstalkSources_855231_30.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/8a048b0a-19f0-42ef-b065-eb26e9997c00_isrForCrosstalkSources_855231_30.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 7d24e776f33416a4cbe53fd5f6b2bce1 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b474a55d-1ab2-48fd-825e-5d7b9f242395_isrForCrosstalkSources_855235_24.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b474a55d-1ab2-48fd-825e-5d7b9f242395_isrForCrosstalkSources_855235_24.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 575778281b6c529b6a61103dbf5171f5 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/a23a0a8d-a6ce-4d10-981d-e5e6e69d4305_isrForCrosstalkSources_855154_13.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/a23a0a8d-a6ce-4d10-981d-e5e6e69d4305_isrForCrosstalkSources_855154_13.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash b6504074260fa90114591b8c71657727 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c93fc277-50f0-400b-a4ff-d3f1c8607b26_isrForCrosstalkSources_855237_7.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c93fc277-50f0-400b-a4ff-d3f1c8607b26_isrForCrosstalkSources_855237_7.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash d1691664e68f3b53e34a4a1a7eb308ab parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/5a848edf-d76a-41a7-ab04-fb004a29be64_isrForCrosstalkSources_855239_11.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/5a848edf-d76a-41a7-ab04-fb004a29be64_isrForCrosstalkSources_855239_11.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 746b70c2c0a3c8aab47d13af651f6852 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1982c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914608 proc_lsst.multi INFO: [multi] job_id 16914608 2024-03-08 10:00:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 10:00:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914608 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914608 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16914608 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-08 10:00:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/eb142208-4265-4c58-8454-67281af9c8bb_isrForCrosstalkSources_855235_3.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/eb142208-4265-4c58-8454-67281af9c8bb_isrForCrosstalkSources_855235_3.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 04ea89eff61d6191566bd4c74ddd04d2 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d26345f-0ae7-461d-82fc-89bacc7712c9_isrForCrosstalkSources_855154_26.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d26345f-0ae7-461d-82fc-89bacc7712c9_isrForCrosstalkSources_855154_26.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 1fd57a494b1a38a2e07bc31be3315b1d parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/410bbf78-3374-4da4-9444-c613f7e8be8c_isrForCrosstalkSources_855236_42.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/410bbf78-3374-4da4-9444-c613f7e8be8c_isrForCrosstalkSources_855236_42.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 25f55da4768ebd8a0a9daaf6131a11aa parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7b203558-f083-40d9-934c-84badd48ecd6_isrForCrosstalkSources_855236_19.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7b203558-f083-40d9-934c-84badd48ecd6_isrForCrosstalkSources_855236_19.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 0309c7bb8849503582053743aacd7de1 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1984a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/798db830-1173-4934-97aa-e06925677a02_isrForCrosstalkSources_855232_2.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/798db830-1173-4934-97aa-e06925677a02_isrForCrosstalkSources_855232_2.stderr parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 40526fd9d62832d66c3a93a3a7d15c69 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7eb89f6b-36d9-4f03-a583-72d9c79bc38a_isrForCrosstalkSources_855237_21.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7eb89f6b-36d9-4f03-a583-72d9c79bc38a_isrForCrosstalkSources_855237_21.stderr parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 0db4bbb862244485de0acf42c513970e parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1980e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/919a4729-d589-4adc-8563-86d8327aacbf_isrForCrosstalkSources_855236_36.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/919a4729-d589-4adc-8563-86d8327aacbf_isrForCrosstalkSources_855236_36.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash 63fed7803875dcbcdccd4bbfdf2dab16 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b715c15e-3707-4373-b9c3-2ea7efa927fb_isrForCrosstalkSources_855241_12.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b715c15e-3707-4373-b9c3-2ea7efa927fb_isrForCrosstalkSources_855241_12.stderr parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 0dd05e83b16ee8ec156bb4064e5b99f3 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a87aa01b-3643-4bb7-aafb-42f34825ade5_isrForCrosstalkSources_855242_59.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a87aa01b-3643-4bb7-aafb-42f34825ade5_isrForCrosstalkSources_855242_59.stderr parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash c89a9aec7081cd713ed6cd4e1d00f9e4 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1985e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/9e23c7be-f606-4d0c-abb7-c8cda0a45b40_isrForCrosstalkSources_855241_7.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/9e23c7be-f606-4d0c-abb7-c8cda0a45b40_isrForCrosstalkSources_855241_7.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash fdadb7e0bd2334dc0e703ed77746af7e parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/55d59b14-cf0d-4b32-abbb-88fe8306fa24_isrForCrosstalkSources_855154_1.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/55d59b14-cf0d-4b32-abbb-88fe8306fa24_isrForCrosstalkSources_855154_1.stderr parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 2d3469deeb680ddd912a7ff6747db042 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a0000f44-c1cf-4c73-bb3a-d22fbfde680a_isrForCrosstalkSources_855231_19.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a0000f44-c1cf-4c73-bb3a-d22fbfde680a_isrForCrosstalkSources_855231_19.stderr parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 5a6c7a767cf67ce94a25f97ebd79ed10 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14e7f5ca-eeca-45d2-b3f9-9c11af3df53b_isrForCrosstalkSources_855240_23.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14e7f5ca-eeca-45d2-b3f9-9c11af3df53b_isrForCrosstalkSources_855240_23.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash c718d63dc9d68c6ea39dfedc637ff103 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1989a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0c97fd68-fff1-436f-9704-4fd469ecc057_isrForCrosstalkSources_855155_60.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0c97fd68-fff1-436f-9704-4fd469ecc057_isrForCrosstalkSources_855155_60.stderr parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash e21c41a90249cf13e38d37ef4000cde9 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1987c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/6f78584e-d723-4b55-a558-ff17d5e2690e_isrForCrosstalkSources_855234_10.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/6f78584e-d723-4b55-a558-ff17d5e2690e_isrForCrosstalkSources_855234_10.stderr parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 3c675acde94f1f78e5b4dd72d7274df1 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/099d780e-6573-474a-8d14-7e6a6b88e980_isrForCrosstalkSources_855233_35.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/099d780e-6573-474a-8d14-7e6a6b88e980_isrForCrosstalkSources_855233_35.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 00e0490c4c20aa7db5f64c5eee95e82e parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/94218fd2-7715-491e-85b2-0c871cca2ac3_isrForCrosstalkSources_855242_46.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/94218fd2-7715-491e-85b2-0c871cca2ac3_isrForCrosstalkSources_855242_46.stderr parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 83429793362c26d63e4eaedf0d2a8178 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/29249387-c04b-4323-9913-9782cba152b6_isrForCrosstalkSources_855154_58.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/29249387-c04b-4323-9913-9782cba152b6_isrForCrosstalkSources_855154_58.stderr parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 6f0dc00a0b55e39b86485cd48b33ce72 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/acfc4072-4f17-4d38-9c17-67df147e79ed_isrForCrosstalkSources_855154_23.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/acfc4072-4f17-4d38-9c17-67df147e79ed_isrForCrosstalkSources_855154_23.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 95d3d9ab85d996591a2c69d92e73332d parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/aa865f4c-f897-415e-895c-8908315f08c3_isrForCrosstalkSources_855237_30.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/aa865f4c-f897-415e-895c-8908315f08c3_isrForCrosstalkSources_855237_30.stderr parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 5a8f8c7ef1411ab7572c7054593398ed parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/93b62b73-079f-465d-af84-3893af5f48d0_isrForCrosstalkSources_855154_16.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/93b62b73-079f-465d-af84-3893af5f48d0_isrForCrosstalkSources_855154_16.stderr parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 54a450114b660d40daf82082f625d6d1 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/11131618-c2cf-4853-a928-396c4b09014b_isrForCrosstalkSources_855233_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/11131618-c2cf-4853-a928-396c4b09014b_isrForCrosstalkSources_855233_60.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash e36495ac2c6777e7be4ffcc86efc4eab parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/29d7ade1-5731-4a6e-bba1-c87d045a106e_isrForCrosstalkSources_855156_52.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/29d7ade1-5731-4a6e-bba1-c87d045a106e_isrForCrosstalkSources_855156_52.stderr parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash bd11c6fdf1ad7e6b59c23e7f5abd3d6b parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1991c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d5217d6-958c-4e4e-b368-2c79d68d0952_isrForCrosstalkSources_855237_4.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d5217d6-958c-4e4e-b368-2c79d68d0952_isrForCrosstalkSources_855237_4.stderr parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash d53fc8698b80bea9c3954e56b52d5a7b parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/979b0487-34fa-4bf8-8f31-5f421eb23c08_isrForCrosstalkSources_855234_54.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/979b0487-34fa-4bf8-8f31-5f421eb23c08_isrForCrosstalkSources_855234_54.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 981d890c1b937694b736fa9af8dc0345 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d8313e35-cbd5-4f5d-bd4b-9abce552e0d1_isrForCrosstalkSources_855231_46.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d8313e35-cbd5-4f5d-bd4b-9abce552e0d1_isrForCrosstalkSources_855231_46.stderr parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 8b54c2a23c0426baf42386155548561d parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9ae5ce35-5153-48ca-ade7-679d3e65657a_isrForCrosstalkSources_855233_22.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9ae5ce35-5153-48ca-ade7-679d3e65657a_isrForCrosstalkSources_855233_22.stderr parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash b9d215fd7692feabf79689efde80aca0 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/6b0640be-eb69-475c-811f-62a1a2d7f0fa_isrForCrosstalkSources_855237_5.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/6b0640be-eb69-475c-811f-62a1a2d7f0fa_isrForCrosstalkSources_855237_5.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash 365b5bf82a302ca08912b7527210f31b parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4924b476-7e29-4345-9c07-45f811f519ba_isrForCrosstalkSources_855238_60.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4924b476-7e29-4345-9c07-45f811f519ba_isrForCrosstalkSources_855238_60.stderr parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 4067d9df4ab0f8e05480d798dcb61a51 parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914609 proc_lsst.multi INFO: [multi] job_id 16914609 2024-03-08 10:00:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 10:00:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914609 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914609 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16914609 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-08 10:00:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/29a74a82-4cb5-4b44-ba11-47c0321649cb_isrForCrosstalkSources_855242_57.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/29a74a82-4cb5-4b44-ba11-47c0321649cb_isrForCrosstalkSources_855242_57.stderr parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash fdb2ed50c8bec6552708ec8b0b6417cc parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1994e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a91b831-ff84-4498-8113-504373243729_isrForCrosstalkSources_855235_39.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a91b831-ff84-4498-8113-504373243729_isrForCrosstalkSources_855235_39.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash a97afbdc2d2c9951aabf3411d19d3455 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1993a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9c51466c-d80e-44d4-8a4c-dc6d2657d538_isrForCrosstalkSources_855239_1.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9c51466c-d80e-44d4-8a4c-dc6d2657d538_isrForCrosstalkSources_855239_1.stderr parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 82d7968ed112588c04282e1ab47cdb29 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1996c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/085896b7-f9ff-4ab3-be86-6c75cf88fe63_isrForCrosstalkSources_855231_49.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/085896b7-f9ff-4ab3-be86-6c75cf88fe63_isrForCrosstalkSources_855231_49.stderr parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 8a80628ff972b4413c9cbf11df11bcaf parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1198ea1c-20fd-4c7a-aa1e-7d189e2ba82f_isrForCrosstalkSources_855232_34.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1198ea1c-20fd-4c7a-aa1e-7d189e2ba82f_isrForCrosstalkSources_855232_34.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 6294cc3fbd89966107c49ecb4d8b56ee parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e316369f-f0b6-4a02-8ee0-e96e459a83bb_isrForCrosstalkSources_855238_59.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e316369f-f0b6-4a02-8ee0-e96e459a83bb_isrForCrosstalkSources_855238_59.stderr parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 8c296491df6d6ccd6cd09e937ae0fbf8 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7ebd90f3-bb64-4dac-8876-dc9c72d9a688_isrForCrosstalkSources_855155_24.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7ebd90f3-bb64-4dac-8876-dc9c72d9a688_isrForCrosstalkSources_855155_24.stderr parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash c1dfa3e5606ba37734414c5de607c7d2 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6ffe80f3-22a0-4321-bb3c-0827b16e0ae2_isrForCrosstalkSources_855238_26.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6ffe80f3-22a0-4321-bb3c-0827b16e0ae2_isrForCrosstalkSources_855238_26.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash d6f14f4f6a2e5adf28f066f21a4a1d2c parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec198900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/501f4bd8-dac8-4768-9ecf-feea2247f0b5_isrForCrosstalkSources_855156_30.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/501f4bd8-dac8-4768-9ecf-feea2247f0b5_isrForCrosstalkSources_855156_30.stderr parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 965385e39437fd5d26ca82b1d4e71037 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12e39556-d7ab-43fc-95d2-35b7e7611429_isrForCrosstalkSources_855231_58.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12e39556-d7ab-43fc-95d2-35b7e7611429_isrForCrosstalkSources_855231_58.stderr parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 615227831548f13fa2c749fd2098863b parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ceee6a31-aa30-4e15-b297-68abfbc5f334_isrForCrosstalkSources_855242_1.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ceee6a31-aa30-4e15-b297-68abfbc5f334_isrForCrosstalkSources_855242_1.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 5259473110f9c5dbc0bd12f96b3073f3 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8b8f4273-99ef-42f5-a51e-e48a23f7a445_isrForCrosstalkSources_855232_52.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8b8f4273-99ef-42f5-a51e-e48a23f7a445_isrForCrosstalkSources_855232_52.stderr parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 7e1cf032a1389ee55515c1cc8d30fe18 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e425db60-adf0-4709-9639-18b19ad64f73_isrForCrosstalkSources_855237_8.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e425db60-adf0-4709-9639-18b19ad64f73_isrForCrosstalkSources_855237_8.stderr parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash f440cbefdfdef73bbd059f96b18d1e19 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/501d1766-6943-446d-b2cd-6d7df7251098_isrForCrosstalkSources_855232_9.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/501d1766-6943-446d-b2cd-6d7df7251098_isrForCrosstalkSources_855232_9.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash c42829037a408c20ccd5500568b9e696 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1999e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bf4deac-acc2-4221-8fdc-f0d766de0b04_isrForCrosstalkSources_855231_42.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bf4deac-acc2-4221-8fdc-f0d766de0b04_isrForCrosstalkSources_855231_42.stderr parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash e32f893ce801de25ca1b87b5a68060f0 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6f6591d6-a1b0-4d25-9f44-e42fb69262dd_isrForCrosstalkSources_855233_18.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6f6591d6-a1b0-4d25-9f44-e42fb69262dd_isrForCrosstalkSources_855233_18.stderr parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash fb104b6931494adb97f0d1588376189e parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/95dc7636-ec93-4419-a4ab-aac6ef153337_isrForCrosstalkSources_855155_19.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/95dc7636-ec93-4419-a4ab-aac6ef153337_isrForCrosstalkSources_855155_19.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 3b25d53defc5a36a4343b91911051dc4 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/2c697529-4393-410c-bdd3-3c2a32c593b8_isrForCrosstalkSources_855233_6.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/2c697529-4393-410c-bdd3-3c2a32c593b8_isrForCrosstalkSources_855233_6.stderr parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 8dbb0490e72f5282652a2b3c97cdd57f parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b20d63d-c126-4ef9-9454-79fa30a74c50_isrForCrosstalkSources_855241_17.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b20d63d-c126-4ef9-9454-79fa30a74c50_isrForCrosstalkSources_855241_17.stderr parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 416aa8bbefb1f5ad2203cf1a68675544 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b858442b-0a67-4053-8f56-f54db0311962_isrForCrosstalkSources_855241_51.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b858442b-0a67-4053-8f56-f54db0311962_isrForCrosstalkSources_855241_51.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 4f239d9ea553273611771d25aaed7b41 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ec170161-85b1-488c-80f4-6d98dde8c795_isrForCrosstalkSources_855239_21.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ec170161-85b1-488c-80f4-6d98dde8c795_isrForCrosstalkSources_855239_21.stderr parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 948a330f476e94869d7f5c28f63f6d4c parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/e67bc7ed-e7b6-48d9-9ee6-d085b540c52e_isrForCrosstalkSources_855234_38.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/e67bc7ed-e7b6-48d9-9ee6-d085b540c52e_isrForCrosstalkSources_855234_38.stderr parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 0d987fec580ea6744d9da2cdbe3aa03f parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/78e62b4c-4a7a-4811-bcd7-2b21d2458b88_isrForCrosstalkSources_855242_53.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/78e62b4c-4a7a-4811-bcd7-2b21d2458b88_isrForCrosstalkSources_855242_53.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 01756181441ca978b80ef39aeaf4c514 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24c7532d-bf87-457f-a73b-bf7d7f2a7034_isrForCrosstalkSources_855231_13.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24c7532d-bf87-457f-a73b-bf7d7f2a7034_isrForCrosstalkSources_855231_13.stderr parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash f6faae81855009eefcfd3dbb7cd651f4 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3fcfc5b-1599-42c6-84b6-aee432815de0_isrForCrosstalkSources_855236_61.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3fcfc5b-1599-42c6-84b6-aee432815de0_isrForCrosstalkSources_855236_61.stderr parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 5f6baede5c6b7a1ca728ea8de3a21f16 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/acf94272-d692-422d-939f-f4f2ca46c2b1_isrForCrosstalkSources_855239_44.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/acf94272-d692-422d-939f-f4f2ca46c2b1_isrForCrosstalkSources_855239_44.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash cbc03fcf768911d47de76991cb3c9a62 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d397ee31-d81a-494a-b3b4-a99e5c5c61c0_isrForCrosstalkSources_855241_38.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d397ee31-d81a-494a-b3b4-a99e5c5c61c0_isrForCrosstalkSources_855241_38.stderr parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 719520c749914d092cc12ba777ca280a parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/31f78770-896e-4801-a35c-7af5fde1a431_isrForCrosstalkSources_855238_42.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/31f78770-896e-4801-a35c-7af5fde1a431_isrForCrosstalkSources_855238_42.stderr parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 73b582df515419cf47330253e854e387 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/377c5e19-1101-40b0-9e9a-90b29f167ebb_isrForCrosstalkSources_855242_3.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/377c5e19-1101-40b0-9e9a-90b29f167ebb_isrForCrosstalkSources_855242_3.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash f6d5abbb09ac231028259e3cb26eb5e4 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/47f54081-8311-4eff-b741-15ac14dc1c41_isrForCrosstalkSources_855154_28.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/47f54081-8311-4eff-b741-15ac14dc1c41_isrForCrosstalkSources_855154_28.stderr parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash b4a0eeb6a1b1002f4d2c8e7b61c4d6b0 parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/23ae707c-f85d-4644-9558-5c6d46e8c223_isrForCrosstalkSources_855231_32.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/23ae707c-f85d-4644-9558-5c6d46e8c223_isrForCrosstalkSources_855231_32.stderr parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 7e46ac2a009c0400ecdc58d1b1cafddc parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67be035e-7f76-4e7e-bc10-a85a0e191841_isrForCrosstalkSources_855239_6.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67be035e-7f76-4e7e-bc10-a85a0e191841_isrForCrosstalkSources_855239_6.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 079395a087d95f29b3128868bd4d6532 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e0557248-630c-4920-a34f-13340501e317_isrForCrosstalkSources_855238_25.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e0557248-630c-4920-a34f-13340501e317_isrForCrosstalkSources_855238_25.stderr parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash f55adae8536e46849eedf7a842edabf2 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c29be1e-fe90-4080-9939-6cff3b538834_isrForCrosstalkSources_855241_39.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c29be1e-fe90-4080-9939-6cff3b538834_isrForCrosstalkSources_855241_39.stderr parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash b16d7441212a19d35f6a4234c9cd0b3e parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/066b5700-44e8-4862-bcaf-1c84a136a142_isrForCrosstalkSources_855234_11.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/066b5700-44e8-4862-bcaf-1c84a136a142_isrForCrosstalkSources_855234_11.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 01f309efeff832595e99076853af7eed parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/768d4231-35bb-44ff-8b4d-09abfac42442_isrForCrosstalkSources_855231_16.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/768d4231-35bb-44ff-8b4d-09abfac42442_isrForCrosstalkSources_855231_16.stderr parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash d620d60e9d58c88905ab60d1dd6da051 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/efc84dee-948a-4dcc-bb70-057c2a1315bb_isrForCrosstalkSources_855232_25.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/efc84dee-948a-4dcc-bb70-057c2a1315bb_isrForCrosstalkSources_855232_25.stderr parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 20c28f8c6e3bea61c3c3a54a3b21be3a parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f9a4c428-5457-44ff-b377-d34e3e8f7420_isrForCrosstalkSources_855156_43.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f9a4c428-5457-44ff-b377-d34e3e8f7420_isrForCrosstalkSources_855156_43.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 0d74c9f358ed6f2f77d9efb5f6f16268 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914610 proc_lsst.multi INFO: [multi] job_id 16914610 2024-03-08 10:00:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 10:00:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914610 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914610 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16914610 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-08 10:00:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c3e55535-6b7e-4f79-afca-70d54ff97860_isrForCrosstalkSources_855154_14.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c3e55535-6b7e-4f79-afca-70d54ff97860_isrForCrosstalkSources_855154_14.stderr parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 66bcd50684f999cdb5fa21c48586f586 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b035e45d-60a3-4173-b84e-28d90f7d2093_isrForCrosstalkSources_855241_53.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b035e45d-60a3-4173-b84e-28d90f7d2093_isrForCrosstalkSources_855241_53.stderr parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash b64264577caed548cf029811815df8e2 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e8ae09dd-8e35-4fe7-9926-f2bda87bccea_isrForCrosstalkSources_855238_35.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e8ae09dd-8e35-4fe7-9926-f2bda87bccea_isrForCrosstalkSources_855238_35.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 12a9c132f7738c42e1ad4e7478b8cea2 parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ff12bda2-fc6b-4c16-97f8-0e1ada968367_isrForCrosstalkSources_855236_7.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ff12bda2-fc6b-4c16-97f8-0e1ada968367_isrForCrosstalkSources_855236_7.stderr parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 5d1b1dd3cc9216a13c838c2f3fc51db3 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/386e3382-b042-4f99-b14c-ee6860587787_isrForCrosstalkSources_855234_32.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/386e3382-b042-4f99-b14c-ee6860587787_isrForCrosstalkSources_855234_32.stderr parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 0961ea11f330c5b1bab768e593c755cb parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fe2dac1f-b3c0-4d3d-84a9-27aac4d17a3e_isrForCrosstalkSources_855235_38.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fe2dac1f-b3c0-4d3d-84a9-27aac4d17a3e_isrForCrosstalkSources_855235_38.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash 9aea70654d3cd207c90cb8a62fdeece5 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/daf21744-aa88-44a5-b63e-8956b8281709_isrForCrosstalkSources_855240_12.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/daf21744-aa88-44a5-b63e-8956b8281709_isrForCrosstalkSources_855240_12.stderr parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 1f3478bfc7b814286b4b8f4eac5069e7 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec13f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2be9391f-4428-4d81-af61-ada8b9b6a855_isrForCrosstalkSources_855242_25.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2be9391f-4428-4d81-af61-ada8b9b6a855_isrForCrosstalkSources_855242_25.stderr parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 72194c6ad5eb860ad6e9fe1a6a6a6ed8 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a7ced4f2-fd17-4061-b874-858572c66138_isrForCrosstalkSources_855156_31.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a7ced4f2-fd17-4061-b874-858572c66138_isrForCrosstalkSources_855156_31.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 5159ded7da463a52e9f1b526b4f1d4c7 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd367a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/aad694cd-adf3-4e91-a912-84e4b07d8fb6_isrForCrosstalkSources_855155_14.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/aad694cd-adf3-4e91-a912-84e4b07d8fb6_isrForCrosstalkSources_855155_14.stderr parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 686721e184f37910d761011a0b7f8ff5 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56e409ea-9eb0-4cbd-9434-c83c958c3673_isrForCrosstalkSources_855237_29.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56e409ea-9eb0-4cbd-9434-c83c958c3673_isrForCrosstalkSources_855237_29.stderr parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 04a1cab5bb3adf9cb7ca2643a482b0d2 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/cf9ccfd6-c407-40a2-b494-a2d584e2272d_isrForCrosstalkSources_855234_34.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/cf9ccfd6-c407-40a2-b494-a2d584e2272d_isrForCrosstalkSources_855234_34.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash c95e532c8ca5f11902306bfb76f499f8 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd365c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/29d0d8c1-4ace-4c9e-8e43-a7da6383bf78_isrForCrosstalkSources_855239_32.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/29d0d8c1-4ace-4c9e-8e43-a7da6383bf78_isrForCrosstalkSources_855239_32.stderr parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash c95fad40275e03e131f0f862c624f32d parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f26822f5-cf30-463e-8bc0-707c335afa49_isrForCrosstalkSources_855232_62.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f26822f5-cf30-463e-8bc0-707c335afa49_isrForCrosstalkSources_855232_62.stderr parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 5ba49c52d385ae4907acdd2d9177fcb5 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd363e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/430d8c6e-98c2-4db0-8a93-cfe983f5b6e5_isrForCrosstalkSources_855233_51.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/430d8c6e-98c2-4db0-8a93-cfe983f5b6e5_isrForCrosstalkSources_855233_51.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 95977fd13a9dd690690167bfdf1af44a parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/1d4292f4-4eea-4fe3-8858-bd06614e0d56_isrForCrosstalkSources_855154_61.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/1d4292f4-4eea-4fe3-8858-bd06614e0d56_isrForCrosstalkSources_855154_61.stderr parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 5a07e7324efca7ef7fb93e1d5bed5362 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12122a40-e26f-4d1d-88ee-29396ce8fb90_isrForCrosstalkSources_855231_56.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12122a40-e26f-4d1d-88ee-29396ce8fb90_isrForCrosstalkSources_855231_56.stderr parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash d7930fb6731ff290e1e35c723d62ac79 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/26530db8-60ce-4127-a8bd-259c2472f33f_isrForCrosstalkSources_855231_10.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/26530db8-60ce-4127-a8bd-259c2472f33f_isrForCrosstalkSources_855231_10.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash 545a61d86f11a2d3513ed82a65d3110d parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6496040-4ac9-42e2-b4d6-174336ebcd3c_isrForCrosstalkSources_855240_5.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6496040-4ac9-42e2-b4d6-174336ebcd3c_isrForCrosstalkSources_855240_5.stderr parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash eaa55a2d9d1676f2a7c581a30de462f5 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/abcc1e3b-62fa-4e0a-a154-a3537474f409_isrForCrosstalkSources_855241_37.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/abcc1e3b-62fa-4e0a-a154-a3537474f409_isrForCrosstalkSources_855241_37.stderr parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash b482cd2ed6ac48358014b1bf5d4144eb parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b23d8cb0-7b14-4462-b06a-35d5cd54c946_isrForCrosstalkSources_855239_31.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b23d8cb0-7b14-4462-b06a-35d5cd54c946_isrForCrosstalkSources_855239_31.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 7dc78f05f1488e1db4aa8e6d1a6cf5b0 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ce7f644-d681-4d2e-ae3a-23dfebf79afc_isrForCrosstalkSources_855236_24.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ce7f644-d681-4d2e-ae3a-23dfebf79afc_isrForCrosstalkSources_855236_24.stderr parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 0970026f2bd91337bb84409de1be4410 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/cad3f9e6-82d3-4fe0-8ea8-cb7482394d51_isrForCrosstalkSources_855156_49.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/cad3f9e6-82d3-4fe0-8ea8-cb7482394d51_isrForCrosstalkSources_855156_49.stderr parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash fb65c9ac757d75819c84c5ff5f934796 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8f47e77c-ca05-46c6-b573-561b698f8e35_isrForCrosstalkSources_855238_32.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8f47e77c-ca05-46c6-b573-561b698f8e35_isrForCrosstalkSources_855238_32.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 266cffb8eac845741aadcef3b1bc62ab parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/636a06dd-85ca-40f9-864c-2db4822e6cae_isrForCrosstalkSources_855237_17.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/636a06dd-85ca-40f9-864c-2db4822e6cae_isrForCrosstalkSources_855237_17.stderr parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 0a72f262650f57f4d59adb41c7b57043 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20d3f772-e8cc-4670-8896-32169ae1310a_isrForCrosstalkSources_855156_17.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20d3f772-e8cc-4670-8896-32169ae1310a_isrForCrosstalkSources_855156_17.stderr parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash b22d0b1277150b3f621da32ba41f2e28 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/cad6f813-b4e8-4dfb-bef2-d56133f5a488_isrForCrosstalkSources_855231_7.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/cad6f813-b4e8-4dfb-bef2-d56133f5a488_isrForCrosstalkSources_855231_7.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 1ad1a78892b0cf8fa62009e1cc94c87e parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7b668df0-55f1-4751-8f22-c7efea07f1f2_isrForCrosstalkSources_855232_60.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7b668df0-55f1-4751-8f22-c7efea07f1f2_isrForCrosstalkSources_855232_60.stderr parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash c709c07cc7192d4862d0d555da7e4d89 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/03712f63-f489-4365-9589-829dc1883971_isrForCrosstalkSources_855240_46.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/03712f63-f489-4365-9589-829dc1883971_isrForCrosstalkSources_855240_46.stderr parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash c7833da8daadb08b7691ef6161194c63 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/7517041e-51ce-418f-90af-eb8baef0d007_isrForCrosstalkSources_855242_41.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/7517041e-51ce-418f-90af-eb8baef0d007_isrForCrosstalkSources_855242_41.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 138eb2b7ae1308bb68bbc10a4bbc15e5 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ab1a2a57-3105-4ee1-aa23-eb643538cd6a_isrForCrosstalkSources_855155_49.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ab1a2a57-3105-4ee1-aa23-eb643538cd6a_isrForCrosstalkSources_855155_49.stderr parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash e1ca281d438eb78c4598a501da7f6b31 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec199800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/36fa73ea-7b32-4561-a7fc-6ec700360988_isrForCrosstalkSources_855238_34.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/36fa73ea-7b32-4561-a7fc-6ec700360988_isrForCrosstalkSources_855238_34.stderr parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 38bb73bd67b31d8cc1acaab8ade4d738 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/416f9eea-7a4c-4bf8-8592-5767ca8a2cd0_isrForCrosstalkSources_855235_42.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/416f9eea-7a4c-4bf8-8592-5767ca8a2cd0_isrForCrosstalkSources_855235_42.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 20516e50846cb6f4f775f4b893725d49 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec1998a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/be9f6d54-6b0d-4aca-8cc5-4e215e24ae22_isrForCrosstalkSources_855233_56.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/be9f6d54-6b0d-4aca-8cc5-4e215e24ae22_isrForCrosstalkSources_855233_56.stderr parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash d5f87cbb45e0bacaa5baf3ea97a8886b parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/63e07620-e0a1-48bc-adc6-6419a2e2c6a9_isrForCrosstalkSources_855154_32.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/63e07620-e0a1-48bc-adc6-6419a2e2c6a9_isrForCrosstalkSources_855154_32.stderr parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 2fc5a5e376989ddc0427d4a8896f1c8b parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/651a8247-62f0-424d-8989-8cbfb0732223_isrForCrosstalkSources_855233_27.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/651a8247-62f0-424d-8989-8cbfb0732223_isrForCrosstalkSources_855233_27.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 68a7c10c3674d3c1f449e1c8de4bd9a4 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80e166a7-2fe9-46bb-9dbe-e3eca0016eaa_isrForCrosstalkSources_855235_55.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80e166a7-2fe9-46bb-9dbe-e3eca0016eaa_isrForCrosstalkSources_855235_55.stderr parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 3ea89b067982c97862bff7e16fb0d1ee parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/57c0f898-4fbf-4c35-84b5-f5a9446da2e8_isrForCrosstalkSources_855238_57.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/57c0f898-4fbf-4c35-84b5-f5a9446da2e8_isrForCrosstalkSources_855238_57.stderr parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash cfb797e2afacd72a8994b70eae37e248 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/214319ec-cdaa-428d-922f-29b4ad933835_isrForCrosstalkSources_855155_61.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/214319ec-cdaa-428d-922f-29b4ad933835_isrForCrosstalkSources_855155_61.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 94c74c167490d3db9e966421930abf73 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd342c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2f3b7344-0c61-4ddf-9664-eb92bff275a4_isrForCrosstalkSources_855239_2.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2f3b7344-0c61-4ddf-9664-eb92bff275a4_isrForCrosstalkSources_855239_2.stderr parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash a0bf3ba941f11edb9355a8e1ef0e2640 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd340e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f412f460-1c47-4c55-a31a-f36069d6b9c4_isrForCrosstalkSources_855236_57.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f412f460-1c47-4c55-a31a-f36069d6b9c4_isrForCrosstalkSources_855236_57.stderr parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 91383ad76dee919fa4aa1cf93991bfa3 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/af133832-1fa2-4a77-bd41-0930d7e890c8_isrForCrosstalkSources_855239_43.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/af133832-1fa2-4a77-bd41-0930d7e890c8_isrForCrosstalkSources_855239_43.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 34ce34a862fc9fbc5cbb8cc4d07bac9f parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ddd3a7da-49a4-48fe-ab01-e6dbd6b82485_isrForCrosstalkSources_855233_5.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ddd3a7da-49a4-48fe-ab01-e6dbd6b82485_isrForCrosstalkSources_855233_5.stderr parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash db8281b478c5b45777534d85a64cc66d parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd344a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f8d184cb-4444-413f-b5cf-1526d681a27b_isrForCrosstalkSources_855233_42.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f8d184cb-4444-413f-b5cf-1526d681a27b_isrForCrosstalkSources_855233_42.stderr parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 81b2cc98c3697e52a67180c5dc58ce1e parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd345e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67f7d643-62d2-49e4-a0b4-5b218e04d57a_isrForCrosstalkSources_855239_51.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67f7d643-62d2-49e4-a0b4-5b218e04d57a_isrForCrosstalkSources_855239_51.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash a7826818c27328e1941eedcab5fbcc16 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/82d10652-90a8-4395-8e6d-cf943278eff4_isrForCrosstalkSources_855232_33.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/82d10652-90a8-4395-8e6d-cf943278eff4_isrForCrosstalkSources_855232_33.stderr parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash c5ea3b2a7ce53564ac191dc262a22383 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d965b5b5-6788-442e-95c2-4cd8fe820317_isrForCrosstalkSources_855240_50.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d965b5b5-6788-442e-95c2-4cd8fe820317_isrForCrosstalkSources_855240_50.stderr parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 8d3c47cf6d8c33f9cb331edc1ab59ab8 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/efe6dc1c-dedf-420e-a371-c3cfbfa8b417_isrForCrosstalkSources_855237_9.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/efe6dc1c-dedf-420e-a371-c3cfbfa8b417_isrForCrosstalkSources_855237_9.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 85b94a8301cda730691243f7ea9df6b7 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd347c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/278400eb-6ea6-4e75-8653-886c9fb51914_isrForCrosstalkSources_855238_12.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/278400eb-6ea6-4e75-8653-886c9fb51914_isrForCrosstalkSources_855238_12.stderr parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash ec3507207ed14e681b226deacc2fb9f6 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/075e741f-d06b-4eb2-85da-f79cfeab1736_isrForCrosstalkSources_855233_46.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/075e741f-d06b-4eb2-85da-f79cfeab1736_isrForCrosstalkSources_855233_46.stderr parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 61c9401bde0ed0f743bdb71024714caf parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ac1b67f8-6b3b-42c0-861a-f493de4be69e_isrForCrosstalkSources_855238_14.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ac1b67f8-6b3b-42c0-861a-f493de4be69e_isrForCrosstalkSources_855238_14.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash c75f95a840d68541e9bff79ac166b649 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd349a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8c35c126-2ff8-4f35-8276-2ef041660b03_isrForCrosstalkSources_855238_48.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8c35c126-2ff8-4f35-8276-2ef041660b03_isrForCrosstalkSources_855238_48.stderr parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 2855acbd34c2f6a026c6db153a3bf2d5 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/62a87044-4d65-4f2f-a595-d927c1687340_isrForCrosstalkSources_855241_4.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/62a87044-4d65-4f2f-a595-d927c1687340_isrForCrosstalkSources_855241_4.stderr parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash d6d14dacf30379b4e4484eb2afee39b6 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0bf31726-8b71-4a80-baa4-a2eeef67a8ce_isrForCrosstalkSources_855155_46.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0bf31726-8b71-4a80-baa4-a2eeef67a8ce_isrForCrosstalkSources_855155_46.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash a16c122a1dcff21fd181ff82ea868faf parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0f90e501-fc04-4d6e-88f8-87832ce6c969_isrForCrosstalkSources_855232_22.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0f90e501-fc04-4d6e-88f8-87832ce6c969_isrForCrosstalkSources_855232_22.stderr parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash dd4f524c796d58557408294eb094b7cf parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f706eda-7ed3-4736-9e34-e63986662329_isrForCrosstalkSources_855241_5.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f706eda-7ed3-4736-9e34-e63986662329_isrForCrosstalkSources_855241_5.stderr parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash da9d12d724eff66c751cdff58a083585 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914611 proc_lsst.multi INFO: [multi] job_id 16914611 2024-03-08 10:00:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:00:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914611 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914611 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16914611 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-08 10:00:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:00:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/503274f4-3164-4df3-9df3-670eba3a8d34_isrForCrosstalkSources_855232_59.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/503274f4-3164-4df3-9df3-670eba3a8d34_isrForCrosstalkSources_855232_59.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash f00e0dfb3eb84a8f9c4730a40c24b3ce parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6699e11d-e6e5-48d7-a6de-b290481fe60f_isrForCrosstalkSources_855232_7.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6699e11d-e6e5-48d7-a6de-b290481fe60f_isrForCrosstalkSources_855232_7.stderr parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 328ed8db7b71452cf18535109c2d53b6 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ca7bf9d2-e28c-48e5-95b0-2992cac1cd86_isrForCrosstalkSources_855155_6.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ca7bf9d2-e28c-48e5-95b0-2992cac1cd86_isrForCrosstalkSources_855155_6.stderr parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash f6197e308d31396cd89467956c75b744 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c0c04287-5319-46a4-b465-03a938526f95_isrForCrosstalkSources_855156_26.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c0c04287-5319-46a4-b465-03a938526f95_isrForCrosstalkSources_855156_26.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash b3bd4e1f82d7c32c232e82b96b5df499 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bef3cfb8-8490-4a71-b7dd-65d966c2492d_isrForCrosstalkSources_855236_41.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bef3cfb8-8490-4a71-b7dd-65d966c2492d_isrForCrosstalkSources_855236_41.stderr parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash a123393d7eb2b86d2c3f74fca0d75afe parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5c54f4ae-f7d3-44ec-b0ee-ecf36b3e3570_isrForCrosstalkSources_855242_32.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5c54f4ae-f7d3-44ec-b0ee-ecf36b3e3570_isrForCrosstalkSources_855242_32.stderr parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash d7ef8631b084fe37f9e0317685a47b25 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1837c797-20c3-45c4-b4dd-ff7124b91ec8_isrForCrosstalkSources_855238_23.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1837c797-20c3-45c4-b4dd-ff7124b91ec8_isrForCrosstalkSources_855238_23.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 2df619f2e5a42a7220dfab627b531cb0 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/292a5b69-0cdf-49a4-84c7-5f2f82b34fc6_isrForCrosstalkSources_855231_50.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/292a5b69-0cdf-49a4-84c7-5f2f82b34fc6_isrForCrosstalkSources_855231_50.stderr parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 49316469af29882588e2e211e039db61 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d7698ada-bb2f-41e1-92c1-0e3ed85cd5b3_isrForCrosstalkSources_855238_17.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d7698ada-bb2f-41e1-92c1-0e3ed85cd5b3_isrForCrosstalkSources_855238_17.stderr parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 6ff072c773581f469d2777ba20d8b879 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1545c96f-f996-4871-8053-2fe9d49fb965_isrForCrosstalkSources_855155_23.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1545c96f-f996-4871-8053-2fe9d49fb965_isrForCrosstalkSources_855155_23.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash c7e4c3c02e0ed0363304504260da6e90 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c5035635-6d99-4e1d-a1c9-5364cff0b3d7_isrForCrosstalkSources_855239_10.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c5035635-6d99-4e1d-a1c9-5364cff0b3d7_isrForCrosstalkSources_855239_10.stderr parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 2aae384899eaad66612ebc97f867b744 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/37dd618e-7ae9-4a02-9d78-1872447be4d6_isrForCrosstalkSources_855236_46.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/37dd618e-7ae9-4a02-9d78-1872447be4d6_isrForCrosstalkSources_855236_46.stderr parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 9d93449bac226bcc5f997c50a0f95d91 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd353a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/83302c21-39fc-4574-81bc-df36d4702135_isrForCrosstalkSources_855154_41.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/83302c21-39fc-4574-81bc-df36d4702135_isrForCrosstalkSources_855154_41.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash a85e2eedaa15755b57c2b25d48efa690 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6c8e0011-15d5-4355-8ec1-44827284b233_isrForCrosstalkSources_855155_4.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6c8e0011-15d5-4355-8ec1-44827284b233_isrForCrosstalkSources_855155_4.stderr parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 7c64dceadd562fb6b74d3b9b1e5587de parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd362a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/df032206-a1cf-45e0-8d03-396f825835c5_isrForCrosstalkSources_855155_28.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/df032206-a1cf-45e0-8d03-396f825835c5_isrForCrosstalkSources_855155_28.stderr parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 3e5377352259da442f3c3362ac882fa9 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd360c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/462f0176-78d5-4b0b-a3cf-9a9f62d3f494_isrForCrosstalkSources_855236_45.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/462f0176-78d5-4b0b-a3cf-9a9f62d3f494_isrForCrosstalkSources_855236_45.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 44eff985df6011a6774e79a41a3417c7 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14b958ac-0a33-4cca-8fd8-e56ccb105eab_isrForCrosstalkSources_855240_6.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14b958ac-0a33-4cca-8fd8-e56ccb105eab_isrForCrosstalkSources_855240_6.stderr parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash bcc1ef5cf661af0fd40e2cdc037f0ee1 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/aec6ba88-e7e3-4b77-9630-fb5f9270391d_isrForCrosstalkSources_855236_1.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/aec6ba88-e7e3-4b77-9630-fb5f9270391d_isrForCrosstalkSources_855236_1.stderr parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash a5e3f27321249db334f316d8d2db3493 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c926047-dbea-49d0-87ed-626f0c616e6f_isrForCrosstalkSources_855154_3.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c926047-dbea-49d0-87ed-626f0c616e6f_isrForCrosstalkSources_855154_3.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash a16bedec17181f32f170bdcf1f46c68e parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03acbb5e-90ca-49c1-b1fb-864c57f9def9_isrForCrosstalkSources_855238_15.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03acbb5e-90ca-49c1-b1fb-864c57f9def9_isrForCrosstalkSources_855238_15.stderr parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 166a024268d2c7e389be5e7f1e844553 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/200953d2-f80d-4c6a-9f96-3031e76259ac_isrForCrosstalkSources_855232_31.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/200953d2-f80d-4c6a-9f96-3031e76259ac_isrForCrosstalkSources_855232_31.stderr parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 20ca163db3e3480db073b390a632bb98 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8b1e9eb6-5543-4cff-8cce-2da64c8df4eb_isrForCrosstalkSources_855234_57.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8b1e9eb6-5543-4cff-8cce-2da64c8df4eb_isrForCrosstalkSources_855234_57.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash d1e6ff5ee0c6b8ed4262810e0b66e1be parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/1f55af03-0b6d-41ae-a571-ca791ccb0f8e_isrForCrosstalkSources_855234_49.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/1f55af03-0b6d-41ae-a571-ca791ccb0f8e_isrForCrosstalkSources_855234_49.stderr parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash f0ddd1c7fda423ebe070ec809e8a27e4 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e446e872-2fa8-4c81-8c1f-f9e2e43a80ca_isrForCrosstalkSources_855241_46.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e446e872-2fa8-4c81-8c1f-f9e2e43a80ca_isrForCrosstalkSources_855241_46.stderr parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 528c3efb11ae2a8154e8fb36073225db parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ee8c9e38-a41b-4a54-b3fb-fcd5e96c6a2f_isrForCrosstalkSources_855240_60.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ee8c9e38-a41b-4a54-b3fb-fcd5e96c6a2f_isrForCrosstalkSources_855240_60.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 3028c70a77f781c36b6f3d6a4dd6fa33 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd359e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3398ed4b-1737-44bc-a84c-096b13329720_isrForCrosstalkSources_855233_49.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3398ed4b-1737-44bc-a84c-096b13329720_isrForCrosstalkSources_855233_49.stderr parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash fb58817a903bd2b87212936cad8cff5f parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/353f74cd-3acf-491d-beb3-5128434c1419_isrForCrosstalkSources_855155_55.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/353f74cd-3acf-491d-beb3-5128434c1419_isrForCrosstalkSources_855155_55.stderr parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash fd67097d67dda63d8f8e104caa7038fd parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/602db23c-d2eb-428f-b6c2-9bf5009aa087_isrForCrosstalkSources_855232_19.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/602db23c-d2eb-428f-b6c2-9bf5009aa087_isrForCrosstalkSources_855232_19.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash cccc42186cd22d71833080d4dd577919 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/924a3291-486f-4f4b-8503-b577f738929e_isrForCrosstalkSources_855154_8.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/924a3291-486f-4f4b-8503-b577f738929e_isrForCrosstalkSources_855154_8.stderr parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 0b2a1ecd37a410a447511888d69ad4e0 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd358a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/bb758d61-a15a-40bc-a334-c783c326070c_isrForCrosstalkSources_855154_56.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/bb758d61-a15a-40bc-a334-c783c326070c_isrForCrosstalkSources_855154_56.stderr parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 161ab555fc799a11fb87f23002ce2a30 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd356c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c49cfd3-d62d-4789-a4e7-44e7196f10bf_isrForCrosstalkSources_855154_60.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c49cfd3-d62d-4789-a4e7-44e7196f10bf_isrForCrosstalkSources_855154_60.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash a20be48e62305df690d6dc6b07bbee93 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d106770b-54a6-4991-aa24-99b11a992860_isrForCrosstalkSources_855238_50.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d106770b-54a6-4991-aa24-99b11a992860_isrForCrosstalkSources_855238_50.stderr parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash ecc47e1d2f18d8edd33257389cff3a36 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/63abfa82-775c-4c17-9286-5c95ce1dac4f_isrForCrosstalkSources_855239_36.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/63abfa82-775c-4c17-9286-5c95ce1dac4f_isrForCrosstalkSources_855239_36.stderr parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 43e7e456965e653637b377c80d6ea622 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ef6b9856-c8ef-42b6-83fc-36a1f7797735_isrForCrosstalkSources_855234_5.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ef6b9856-c8ef-42b6-83fc-36a1f7797735_isrForCrosstalkSources_855234_5.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 4662ea86d7ce7e81d4b1133d4bb57f78 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd354e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914612 proc_lsst.multi INFO: [multi] job_id 16914612 2024-03-08 10:00:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 10:00:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914612 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914612 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16914612 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-08 10:00:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-08 10:00:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 10:00:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/97c8c4de-fe22-455e-abee-06771105220c_isrForCrosstalkSources_855238_21.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/97c8c4de-fe22-455e-abee-06771105220c_isrForCrosstalkSources_855238_21.stderr parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash d293ec19da4c1a1045be7cb44c839f4b parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0df4bde1-c858-4373-ad1e-9222df325988_isrForCrosstalkSources_855155_54.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0df4bde1-c858-4373-ad1e-9222df325988_isrForCrosstalkSources_855155_54.stderr parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 66b28de24092b195616c088fbf5c180c parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5361931b-eff3-4631-82fe-5d9e05282b88_isrForCrosstalkSources_855233_61.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5361931b-eff3-4631-82fe-5d9e05282b88_isrForCrosstalkSources_855233_61.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash d05c7c484d41e3e933c3ccacb74bbf87 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d89a56a5-a814-4069-adc9-21f46c1ec1a4_isrForCrosstalkSources_855232_16.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d89a56a5-a814-4069-adc9-21f46c1ec1a4_isrForCrosstalkSources_855232_16.stderr parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 1104106df43597afa2b69e0254426866 parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd351c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/8dba948e-3a0e-4077-b60f-6a76acb2a14b_isrForCrosstalkSources_855235_47.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/8dba948e-3a0e-4077-b60f-6a76acb2a14b_isrForCrosstalkSources_855235_47.stderr parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 4c1300eab5445cc264153d2c04d17944 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a303ec76-bc78-455e-a1dd-d91a3628e210_isrForCrosstalkSources_855234_45.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a303ec76-bc78-455e-a1dd-d91a3628e210_isrForCrosstalkSources_855234_45.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 3aeafbc77246e9b0b915b611f3ab7808 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/72b0e870-4e18-42eb-a82a-6aa2552e35b5_isrForCrosstalkSources_855242_24.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/72b0e870-4e18-42eb-a82a-6aa2552e35b5_isrForCrosstalkSources_855242_24.stderr parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 7f726cf5884f62b409d7c52c07d0b939 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6c4af414-94f8-4128-a301-e0f25fc7d3cf_isrForCrosstalkSources_855238_53.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6c4af414-94f8-4128-a301-e0f25fc7d3cf_isrForCrosstalkSources_855238_53.stderr parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash a36369c3c189058ce2015ed413fb57de parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4ecbbc3c-5f27-412b-93eb-569c40941bc0_isrForCrosstalkSources_855237_22.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4ecbbc3c-5f27-412b-93eb-569c40941bc0_isrForCrosstalkSources_855237_22.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 2b0966b4ab32da0803a6b2b3a0661a8d parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ca793160-79d7-4be2-a5e8-723bdf77adea_isrForCrosstalkSources_855240_33.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ca793160-79d7-4be2-a5e8-723bdf77adea_isrForCrosstalkSources_855240_33.stderr parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 309d95b2e383db1667622c8d9b3c63b6 parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f34faa7a-2202-4293-84fb-91cbc43a585a_isrForCrosstalkSources_855232_13.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f34faa7a-2202-4293-84fb-91cbc43a585a_isrForCrosstalkSources_855232_13.stderr parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 8f547a326aefe9d3bbc4ca9b06180d42 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e4960ed0-8082-4307-9c46-b63ddf58b4bd_isrForCrosstalkSources_855238_61.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e4960ed0-8082-4307-9c46-b63ddf58b4bd_isrForCrosstalkSources_855238_61.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash a486878565b376798a0362718981924b parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5f2c6fcc-5224-457d-9c4f-04097512d12e_isrForCrosstalkSources_855236_13.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5f2c6fcc-5224-457d-9c4f-04097512d12e_isrForCrosstalkSources_855236_13.stderr parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 640df3e6afe7c08a6c0b045da55932b3 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c9209bca-2068-4f38-85a9-db1aa5148b3d_isrForCrosstalkSources_855239_41.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c9209bca-2068-4f38-85a9-db1aa5148b3d_isrForCrosstalkSources_855239_41.stderr parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 505aa58914c78bddc247032940ea0f83 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d7340b05-6e11-4758-8bd2-781054a63a37_isrForCrosstalkSources_855241_40.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d7340b05-6e11-4758-8bd2-781054a63a37_isrForCrosstalkSources_855241_40.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 7b7179e119698d354100fe4fab952cbd parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/209c5818-3f93-4b5d-adf3-2be2a545d239_isrForCrosstalkSources_855235_23.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/209c5818-3f93-4b5d-adf3-2be2a545d239_isrForCrosstalkSources_855235_23.stderr parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash e9386abb98f61766a790e88ed5cfbc9a parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/31f3de11-5fc0-4746-9754-d11d32bcb2f7_isrForCrosstalkSources_855236_27.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/31f3de11-5fc0-4746-9754-d11d32bcb2f7_isrForCrosstalkSources_855236_27.stderr parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash 677a17c96b6f79d04fc3971bf1d25c0d parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c257b6a1-5182-4ec5-8e37-00ab82a456dc_isrForCrosstalkSources_855232_58.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c257b6a1-5182-4ec5-8e37-00ab82a456dc_isrForCrosstalkSources_855232_58.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash a7caa5ecd30b7736f0d922d291df31fd parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/397693f6-086a-4ce2-bb6c-9e65507bee18_isrForCrosstalkSources_855154_42.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/397693f6-086a-4ce2-bb6c-9e65507bee18_isrForCrosstalkSources_855154_42.stderr parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 7c06c3dfcc48f344af483938f29edeef parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a75155b2-ee31-4f21-bc9e-1740c1212407_isrForCrosstalkSources_855239_52.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a75155b2-ee31-4f21-bc9e-1740c1212407_isrForCrosstalkSources_855239_52.stderr parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash c971017c0af38a2f6c6254f180a6770d parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/caaed011-6586-4d1e-b07c-1d7162994416_isrForCrosstalkSources_855154_27.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/caaed011-6586-4d1e-b07c-1d7162994416_isrForCrosstalkSources_855154_27.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash c0bd5d9254d74cd900c8bf45fe978728 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9059a1b3-c5a4-48f1-8490-76745b0aa590_isrForCrosstalkSources_855239_49.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9059a1b3-c5a4-48f1-8490-76745b0aa590_isrForCrosstalkSources_855239_49.stderr parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 200e6196c92dafd33f10881ed4691df0 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2f217c32-21a2-49a3-96dd-a09c9ec34819_isrForCrosstalkSources_855154_9.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2f217c32-21a2-49a3-96dd-a09c9ec34819_isrForCrosstalkSources_855154_9.stderr parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash c963744b2539b46ea48980326e75497b parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd372e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/75e49a83-8204-42cd-b5ed-03d63f4866b9_isrForCrosstalkSources_855154_55.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/75e49a83-8204-42cd-b5ed-03d63f4866b9_isrForCrosstalkSources_855154_55.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 2b5adbb10f9d35a3aa4216fefc4cce20 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd371a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ddad05d8-23ea-48d2-b714-f2316005ee85_isrForCrosstalkSources_855231_41.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ddad05d8-23ea-48d2-b714-f2316005ee85_isrForCrosstalkSources_855231_41.stderr parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 35f0936dfe385fe72ba4205dfc10a676 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4889c96f-5c53-4464-97b7-c14341867a1e_isrForCrosstalkSources_855236_31.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4889c96f-5c53-4464-97b7-c14341867a1e_isrForCrosstalkSources_855236_31.stderr parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash cf65aef0ebd68bdb7369aac1d4b4eeda parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f791457d-5ab4-46f5-a2b8-7af34ef4a56c_isrForCrosstalkSources_855240_4.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f791457d-5ab4-46f5-a2b8-7af34ef4a56c_isrForCrosstalkSources_855240_4.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 505a82860610b11510d19cf12f073ead parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd374c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4cc6360a-8282-433f-a960-28e00459bb46_isrForCrosstalkSources_855242_50.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4cc6360a-8282-433f-a960-28e00459bb46_isrForCrosstalkSources_855242_50.stderr parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 3d1bf5439112011d7fcf7020d7507759 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bff693f5-7620-4b3b-9f28-752d44c46b4a_isrForCrosstalkSources_855237_14.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bff693f5-7620-4b3b-9f28-752d44c46b4a_isrForCrosstalkSources_855237_14.stderr parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash fc0e3dd335b37e5d93ec5a9ee0b4bbc2 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3ef22ab7-cac7-4dbd-942c-060b34edcf5b_isrForCrosstalkSources_855155_43.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3ef22ab7-cac7-4dbd-942c-060b34edcf5b_isrForCrosstalkSources_855155_43.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 6a976fdb4412b1a61e4e326111f27ffd parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72c2908c-4dcc-4cd8-b08b-6f4d02a33e7c_isrForCrosstalkSources_855240_29.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72c2908c-4dcc-4cd8-b08b-6f4d02a33e7c_isrForCrosstalkSources_855240_29.stderr parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash cc6740005efa3a23751fe970dea1f190 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914613 proc_lsst.multi INFO: [multi] job_id 16914613 2024-03-08 10:00:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 10:00:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914613 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914613 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16914613 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-08 10:00:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-08 10:00:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 10:00:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1312ce44-4355-4338-afe9-6f4c073fe3a0_isrForCrosstalkSources_855238_3.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1312ce44-4355-4338-afe9-6f4c073fe3a0_isrForCrosstalkSources_855238_3.stderr parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 540df451f3d1867d77fea5a2304afa2a parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/b0a1949a-ab4b-4934-916b-230ec6f3bb9c_isrForCrosstalkSources_855240_32.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/b0a1949a-ab4b-4934-916b-230ec6f3bb9c_isrForCrosstalkSources_855240_32.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 881e722f47e02fb2c6ae7245cbf61eef parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd35300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc11fb06-2ab8-47e2-82de-6a6ab143a61b_isrForCrosstalkSources_855234_51.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc11fb06-2ab8-47e2-82de-6a6ab143a61b_isrForCrosstalkSources_855234_51.stderr parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash a0fd8d5857e907184ab6b1f9edb2387e parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/04ee46fa-d488-48ac-86ae-e72c56d3f80a_isrForCrosstalkSources_855232_43.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/04ee46fa-d488-48ac-86ae-e72c56d3f80a_isrForCrosstalkSources_855232_43.stderr parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash a0e007e1daadb1d92575769f9b1d12e1 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd36980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d27848b1-b532-4944-829d-0cdb6375331c_isrForCrosstalkSources_855240_61.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d27848b1-b532-4944-829d-0cdb6375331c_isrForCrosstalkSources_855240_61.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 09c5d353427595c74527baeaeb2c6fc4 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd379c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/13df86d2-0da7-4c09-85fd-7ba504492524_isrForCrosstalkSources_855233_4.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/13df86d2-0da7-4c09-85fd-7ba504492524_isrForCrosstalkSources_855233_4.stderr parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 32f85ad5caf9009eb3fb3e184b5ce83c parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ec19ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4e275644-e9b9-43b2-b676-5b8ce77e7fcd_isrForCrosstalkSources_855236_32.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4e275644-e9b9-43b2-b676-5b8ce77e7fcd_isrForCrosstalkSources_855236_32.stderr parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 5e9dc660cc35e4b0bbf0917ed3d3ae6a parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80132e35-fe15-416f-925f-0be82fa289db_isrForCrosstalkSources_855235_62.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80132e35-fe15-416f-925f-0be82fa289db_isrForCrosstalkSources_855235_62.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 2a59db2ddf527f222b1c31367d114ffe parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd368e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8446c04d-97d6-430b-904b-903ff9fb84fd_isrForCrosstalkSources_855238_11.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8446c04d-97d6-430b-904b-903ff9fb84fd_isrForCrosstalkSources_855238_11.stderr parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash a3adb254a6a8ed9388ca66af5953024d parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd376a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2a81fde4-c219-4ba3-a85a-a345b5898854_isrForCrosstalkSources_855155_17.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2a81fde4-c219-4ba3-a85a-a345b5898854_isrForCrosstalkSources_855155_17.stderr parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 3bd879881296f63bf1ea9786252aa5ef parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8ea27f4-5d72-4052-b2a7-72b85e343abc_isrForCrosstalkSources_855234_46.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8ea27f4-5d72-4052-b2a7-72b85e343abc_isrForCrosstalkSources_855234_46.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 3172fbd22299e1147ce12cae995b6739 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d8b2c286-8739-4d34-bc3f-8f16f5e6fba5_isrForCrosstalkSources_855241_36.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d8b2c286-8739-4d34-bc3f-8f16f5e6fba5_isrForCrosstalkSources_855241_36.stderr parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 90ab47cf914c9b9e19789ce845ff7d4a parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b39de129-85cd-4775-bb52-e1742602c1a8_isrForCrosstalkSources_855154_7.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b39de129-85cd-4775-bb52-e1742602c1a8_isrForCrosstalkSources_855154_7.stderr parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 15926d1e1b95901d985c37ecaff2e8d0 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3aed75e2-963a-4d22-9097-56e462db2c61_isrForCrosstalkSources_855231_54.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3aed75e2-963a-4d22-9097-56e462db2c61_isrForCrosstalkSources_855231_54.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 2ee74a571aed20e53da38a9a155e4116 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8a1e347-999b-4131-bc91-962ff79ecbcd_isrForCrosstalkSources_855234_28.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8a1e347-999b-4131-bc91-962ff79ecbcd_isrForCrosstalkSources_855234_28.stderr parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 05283f7180051709cbc09711486c0026 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/2f80c4cc-eae3-42c0-95f3-93c518d01776_isrForCrosstalkSources_855156_56.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/2f80c4cc-eae3-42c0-95f3-93c518d01776_isrForCrosstalkSources_855156_56.stderr parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 5b4a51b1753b952b9aa03800393438d9 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d1a613af-2db5-49e7-8c0a-92ed09e3d617_isrForCrosstalkSources_855154_45.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d1a613af-2db5-49e7-8c0a-92ed09e3d617_isrForCrosstalkSources_855154_45.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash a368548162f4f71562d8f00f961e9fe5 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/7d00d7de-cf4c-4b29-bb54-96bdfeb7d41b_isrForCrosstalkSources_855235_21.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/7d00d7de-cf4c-4b29-bb54-96bdfeb7d41b_isrForCrosstalkSources_855235_21.stderr parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 4f55a03cb3ed6aa74b65528f6456f640 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/55961318-52a0-483f-bb4f-5f5e9e246374_isrForCrosstalkSources_855241_59.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/55961318-52a0-483f-bb4f-5f5e9e246374_isrForCrosstalkSources_855241_59.stderr parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash a0b5d414b69202ce0d1b3cb743a04bcf parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0762d972-5188-4037-a42e-c0e1010e5bc7_isrForCrosstalkSources_855234_8.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0762d972-5188-4037-a42e-c0e1010e5bc7_isrForCrosstalkSources_855234_8.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash c763ff80568f6b374595eda63703cc83 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d9b451de-d04c-4738-88a1-28d187057b91_isrForCrosstalkSources_855236_51.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d9b451de-d04c-4738-88a1-28d187057b91_isrForCrosstalkSources_855236_51.stderr parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 43870fca62c559f7698378642e119344 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/da877c33-c23e-4ccd-8fb6-c95d2b025eed_isrForCrosstalkSources_855231_44.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/da877c33-c23e-4ccd-8fb6-c95d2b025eed_isrForCrosstalkSources_855231_44.stderr parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 6849d34aec00b2f44180ffc0dc9fe781 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3dfd6f6f-190c-4d50-bc1b-2599320a22fb_isrForCrosstalkSources_855155_38.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3dfd6f6f-190c-4d50-bc1b-2599320a22fb_isrForCrosstalkSources_855155_38.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 77f09767fe42d5681e42a7cc2d00afd8 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9f8cf1f2-bbd5-41b2-aa87-5c6beab8e7d4_isrForCrosstalkSources_855231_25.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9f8cf1f2-bbd5-41b2-aa87-5c6beab8e7d4_isrForCrosstalkSources_855231_25.stderr parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 0617403645f429564036a75d5d8d0eb4 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2cde778-28a8-4421-b59c-cdc0ab860c5c_isrForCrosstalkSources_855155_26.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2cde778-28a8-4421-b59c-cdc0ab860c5c_isrForCrosstalkSources_855155_26.stderr parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 0083eeda0631556c69d0d9695e6ee738 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/434b97ea-099e-459a-976b-864ad9a9e0b2_isrForCrosstalkSources_855156_61.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/434b97ea-099e-459a-976b-864ad9a9e0b2_isrForCrosstalkSources_855156_61.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash efcbf9fddbd9326951c3a3b139d1df6d parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/944afd90-3d21-431b-9324-6cf5f07a649c_isrForCrosstalkSources_855155_18.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/944afd90-3d21-431b-9324-6cf5f07a649c_isrForCrosstalkSources_855155_18.stderr parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 639ad9a0e99be9cbb6ce373470fb1e82 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b10944c2-4d79-47ec-b5ad-d6157992ce13_isrForCrosstalkSources_855242_43.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b10944c2-4d79-47ec-b5ad-d6157992ce13_isrForCrosstalkSources_855242_43.stderr parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 0f0f275529d9897f8b41b0ea93466bf4 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e94ec665-0247-4168-8626-ab2317f90a26_isrForCrosstalkSources_855237_25.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e94ec665-0247-4168-8626-ab2317f90a26_isrForCrosstalkSources_855237_25.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash f3c5f50f23d40772072e51f9f61edf90 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2e3456a5-fd84-4a19-82e5-fdb025f39ccf_isrForCrosstalkSources_855240_19.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2e3456a5-fd84-4a19-82e5-fdb025f39ccf_isrForCrosstalkSources_855240_19.stderr parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash e0108b27952274104cb8315a05de8539 parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/2c23ea9b-e830-4c54-bdad-b95513d0faf7_isrForCrosstalkSources_855236_37.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/2c23ea9b-e830-4c54-bdad-b95513d0faf7_isrForCrosstalkSources_855236_37.stderr parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash dce32e16f917ae2e3870c7579c30c663 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bc22ebbb-acae-4f76-809a-eff78cfac1c5_isrForCrosstalkSources_855236_59.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bc22ebbb-acae-4f76-809a-eff78cfac1c5_isrForCrosstalkSources_855236_59.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash c4c32b9237576b3e19333f8b8afca7cd parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0cc662d6-be6b-4243-b6d7-15a88e3a8b92_isrForCrosstalkSources_855242_58.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0cc662d6-be6b-4243-b6d7-15a88e3a8b92_isrForCrosstalkSources_855242_58.stderr parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 230f4bb3ec3e1b3244299b2352117a1d parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/da9a0767-d346-4cf4-bb57-7dc5712045d6_isrForCrosstalkSources_855156_25.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/da9a0767-d346-4cf4-bb57-7dc5712045d6_isrForCrosstalkSources_855156_25.stderr parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 9c569e7af173a8d502ffa5c0aebb98ff parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914614 proc_lsst.multi INFO: [multi] job_id 16914614 2024-03-08 10:00:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 10:00:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914614 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914614 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16914614 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-08 10:00:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-08 10:00:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 10:00:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5f56c7a5-b08d-43ad-9f15-85fad645a032_isrForCrosstalkSources_855155_44.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5f56c7a5-b08d-43ad-9f15-85fad645a032_isrForCrosstalkSources_855155_44.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 64f1f2abf2e823d52541098671a5ab69 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd34ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f3cae9a2-2414-4cda-b48f-e1e61fbe6ea7_isrForCrosstalkSources_855233_50.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f3cae9a2-2414-4cda-b48f-e1e61fbe6ea7_isrForCrosstalkSources_855233_50.stderr parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash fff39773655d620052209854e02f785e parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/25e59c92-fad4-47fe-9f75-4fd1258c7dbd_isrForCrosstalkSources_855239_16.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/25e59c92-fad4-47fe-9f75-4fd1258c7dbd_isrForCrosstalkSources_855239_16.stderr parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash 58fe9d5e74af3ccf49b5410f2d630fa4 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd377e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8a12c318-db95-4b68-9ed2-9bbe0923fa55_isrForCrosstalkSources_855238_18.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8a12c318-db95-4b68-9ed2-9bbe0923fa55_isrForCrosstalkSources_855238_18.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 257d4e0286a036aa6bfc47d8dae8ed4c parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/88f1b536-15b7-4055-b9ca-ec87b2d9d2f3_isrForCrosstalkSources_855238_10.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/88f1b536-15b7-4055-b9ca-ec87b2d9d2f3_isrForCrosstalkSources_855238_10.stderr parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 34078ba734d6cb8723b3a1aa94a88110 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/242321fb-223d-48c8-9919-3c99b4d9ef78_isrForCrosstalkSources_855241_11.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/242321fb-223d-48c8-9919-3c99b4d9ef78_isrForCrosstalkSources_855241_11.stderr parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash c2b357bb8e3c921039351ee112c98f69 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a664abce-f5f1-4011-8d0e-14d829a9ed10_isrForCrosstalkSources_855237_49.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a664abce-f5f1-4011-8d0e-14d829a9ed10_isrForCrosstalkSources_855237_49.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 8fc813555557a36f40aacbcba7298ba3 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ec47e380-0de0-40f6-afa3-b2d409476dd5_isrForCrosstalkSources_855234_39.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ec47e380-0de0-40f6-afa3-b2d409476dd5_isrForCrosstalkSources_855234_39.stderr parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash ec5f4c118a275c1111a915b4570b7370 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5143f579-31c5-47de-9a1c-eeb851a9c6bf_isrForCrosstalkSources_855234_35.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5143f579-31c5-47de-9a1c-eeb851a9c6bf_isrForCrosstalkSources_855234_35.stderr parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash f57eff13e5a3197691d556a00af31382 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/494cd584-9775-46ff-a685-a43e68fb85bc_isrForCrosstalkSources_855236_48.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/494cd584-9775-46ff-a685-a43e68fb85bc_isrForCrosstalkSources_855236_48.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash c5e4e8ddbec331f26879f9db535820ca parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6beae20c-d540-40ff-b902-68ccda0f55b6_isrForCrosstalkSources_855238_33.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6beae20c-d540-40ff-b902-68ccda0f55b6_isrForCrosstalkSources_855238_33.stderr parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash d214ba6e9810c6d87244ec3d1e7cd0fc parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8522deea-16c2-4761-8ca3-5e8ef7a7baa4_isrForCrosstalkSources_855237_2.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8522deea-16c2-4761-8ca3-5e8ef7a7baa4_isrForCrosstalkSources_855237_2.stderr parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 9f2a5344240dc5d7f2c9a5c23027f600 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7781fdbe-6df3-422b-925e-5cd09443ec5b_isrForCrosstalkSources_855154_11.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7781fdbe-6df3-422b-925e-5cd09443ec5b_isrForCrosstalkSources_855154_11.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 38df95775682ecf3d2b004ec456fd68e parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5e3b851f-f9ee-4174-9b04-d2b042367a1c_isrForCrosstalkSources_855235_40.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5e3b851f-f9ee-4174-9b04-d2b042367a1c_isrForCrosstalkSources_855235_40.stderr parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 03eaf105d26f9782785c62b4b43822d8 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/fab72498-2c27-44c3-a540-32d7e7a2877f_isrForCrosstalkSources_855156_54.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/fab72498-2c27-44c3-a540-32d7e7a2877f_isrForCrosstalkSources_855156_54.stderr parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 1a052a1c2c7f60948c8b320acfc9f344 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/533ed82d-4247-4d23-ad72-006cc1726ca8_isrForCrosstalkSources_855233_41.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/533ed82d-4247-4d23-ad72-006cc1726ca8_isrForCrosstalkSources_855233_41.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash f7f0c3a13186e4f303f2deb91483c101 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/fd167fcc-6b84-4dc8-bb8f-8542c9eb2757_isrForCrosstalkSources_855233_1.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/fd167fcc-6b84-4dc8-bb8f-8542c9eb2757_isrForCrosstalkSources_855233_1.stderr parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 5be368222ce786cb9940791bf3d4ec6f parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/19b19c96-0d8b-4a1b-ab04-bb15141824fb_isrForCrosstalkSources_855233_23.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/19b19c96-0d8b-4a1b-ab04-bb15141824fb_isrForCrosstalkSources_855233_23.stderr parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 2fff9101b0e74aeba4e50c8440ad2cdc parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/22261aab-5937-4545-8a82-d895c98ccbac_isrForCrosstalkSources_855240_52.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/22261aab-5937-4545-8a82-d895c98ccbac_isrForCrosstalkSources_855240_52.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash cecf92f7b1c9c825e2cca2609d6f7121 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/9254a2af-faa5-4296-987a-a6412d646108_isrForCrosstalkSources_855236_8.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/9254a2af-faa5-4296-987a-a6412d646108_isrForCrosstalkSources_855236_8.stderr parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 53b8ac76e3a50e0a77aa8c0539b6f6bd parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c77f7b27-8225-45bf-a3e2-9ff49e240fb2_isrForCrosstalkSources_855237_47.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c77f7b27-8225-45bf-a3e2-9ff49e240fb2_isrForCrosstalkSources_855237_47.stderr parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 0d3b8ce244d01487c3c18367d6272099 parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3069e4bf-e145-4211-aa83-7f959c39c54d_isrForCrosstalkSources_855155_37.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3069e4bf-e145-4211-aa83-7f959c39c54d_isrForCrosstalkSources_855155_37.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 20812647b64ffbe72bfb85c018c37e2a parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a5a00542-860a-4946-9f2f-570f2b4ebb8e_isrForCrosstalkSources_855239_42.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a5a00542-860a-4946-9f2f-570f2b4ebb8e_isrForCrosstalkSources_855239_42.stderr parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 98c87b17564c6a948f3dc5c89464a0ab parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/002b2d3a-db57-4eb1-90d4-1e706bf48864_isrForCrosstalkSources_855242_39.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/002b2d3a-db57-4eb1-90d4-1e706bf48864_isrForCrosstalkSources_855242_39.stderr parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash ac60ecd8be89090c423aad6a13c51c1c parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a075b46-38e2-47f7-bae7-64123ec6d60d_isrForCrosstalkSources_855237_38.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a075b46-38e2-47f7-bae7-64123ec6d60d_isrForCrosstalkSources_855237_38.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash ca7093eb7d92299527bb2c96e2349710 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/78b77a16-5fe1-4031-9b87-e31913e6fc49_isrForCrosstalkSources_855155_5.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/78b77a16-5fe1-4031-9b87-e31913e6fc49_isrForCrosstalkSources_855155_5.stderr parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 941def814b4ab2ece42b4178ab9074e8 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/30ca7de3-86ce-41eb-bb9c-863b635d468b_isrForCrosstalkSources_855154_39.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/30ca7de3-86ce-41eb-bb9c-863b635d468b_isrForCrosstalkSources_855154_39.stderr parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 00b903c2c3f0cb6390dfd21aea826824 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f21615e7-dd0b-4d9a-bcd6-ea5b4846d0e8_isrForCrosstalkSources_855240_3.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f21615e7-dd0b-4d9a-bcd6-ea5b4846d0e8_isrForCrosstalkSources_855240_3.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 565d6d532c0269af99a70d3dc58ce9ba parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/984f13c8-842f-4e30-879a-04e1529e7456_isrForCrosstalkSources_855235_32.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/984f13c8-842f-4e30-879a-04e1529e7456_isrForCrosstalkSources_855235_32.stderr parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 946860518f2ba04d91978aacb1a6fce5 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/042ac56e-84bb-4008-88c0-182c6814e5bb_isrForCrosstalkSources_855235_37.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/042ac56e-84bb-4008-88c0-182c6814e5bb_isrForCrosstalkSources_855235_37.stderr parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 8ca9c8b6f12fc3761322a51fa7f320af parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/04fa0369-f042-4a67-9c34-7a6e47da52c3_isrForCrosstalkSources_855233_25.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/04fa0369-f042-4a67-9c34-7a6e47da52c3_isrForCrosstalkSources_855233_25.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash d96257717caa3a3e186896dfbf93c831 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5b5f178e-ac1e-4044-b035-bf811b6fb3cb_isrForCrosstalkSources_855155_35.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5b5f178e-ac1e-4044-b035-bf811b6fb3cb_isrForCrosstalkSources_855155_35.stderr parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 76cbfb08edaf6c112e3ce2d584abc42c parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2c6fb920-ddf2-4cd0-9fef-6ec255b16778_isrForCrosstalkSources_855234_43.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2c6fb920-ddf2-4cd0-9fef-6ec255b16778_isrForCrosstalkSources_855234_43.stderr parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash a83f7bb242cda85b450c7625b49db2ee parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/339f9ae9-f86c-4fce-b64a-d23725fabf87_isrForCrosstalkSources_855235_48.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/339f9ae9-f86c-4fce-b64a-d23725fabf87_isrForCrosstalkSources_855235_48.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 0daf87b77f7840cbf10ee49533d23c8c parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a22f498c-694d-4663-9bca-dcaff74a22c8_isrForCrosstalkSources_855232_61.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a22f498c-694d-4663-9bca-dcaff74a22c8_isrForCrosstalkSources_855232_61.stderr parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 4f8d27dbbe7bf76c31497a1ca3cead5a parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e7d49e2c-ca26-41d1-bceb-e7e239df0f46_isrForCrosstalkSources_855231_55.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e7d49e2c-ca26-41d1-bceb-e7e239df0f46_isrForCrosstalkSources_855231_55.stderr parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 2b16fbe052f171a4df72c8dde50eb927 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/58d51597-aa1f-4225-9002-59c61b7ca4ce_isrForCrosstalkSources_855235_2.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/58d51597-aa1f-4225-9002-59c61b7ca4ce_isrForCrosstalkSources_855235_2.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 94bed6aae596adcc455ffd165a51708e parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914615 proc_lsst.multi INFO: [multi] job_id 16914615 2024-03-08 10:00:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 10:00:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914615 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914615 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16914615 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-08 10:00:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-08 10:00:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 10:00:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/fdf6e0de-23ec-4400-8ee1-188a3f85fdcd_isrForCrosstalkSources_855155_56.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/fdf6e0de-23ec-4400-8ee1-188a3f85fdcd_isrForCrosstalkSources_855155_56.stderr parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash feca5965e840da13219593a369921f9c parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9bb57b29-e499-40d4-999d-513bd9b3ef49_isrForCrosstalkSources_855154_31.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9bb57b29-e499-40d4-999d-513bd9b3ef49_isrForCrosstalkSources_855154_31.stderr parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 41c412ae67bf75f50540c67591eade13 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/357a1cf2-7853-417c-bc6e-419ad7a91221_isrForCrosstalkSources_855155_7.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/357a1cf2-7853-417c-bc6e-419ad7a91221_isrForCrosstalkSources_855155_7.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 5e392558b5dc4fcfcb6ef54b1cd1f1d4 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0ca8f778-37c9-4c91-84b3-3a5c778c4988_isrForCrosstalkSources_855232_1.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0ca8f778-37c9-4c91-84b3-3a5c778c4988_isrForCrosstalkSources_855232_1.stderr parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 46750c56180135c4ee4602f01abdc9ca parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c7d602f7-4fa2-4022-bebb-5d61230614d2_isrForCrosstalkSources_855242_19.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c7d602f7-4fa2-4022-bebb-5d61230614d2_isrForCrosstalkSources_855242_19.stderr parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash ef0c1f8c6068250558b2fd861e587a16 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfd37880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3a49c34f-528c-4535-a362-e41ecdae145f_isrForCrosstalkSources_855234_62.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3a49c34f-528c-4535-a362-e41ecdae145f_isrForCrosstalkSources_855234_62.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash c2d34447d103fbe092b8127fa9bc4507 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/75d8db9e-a409-42ec-a678-d4cebfb35698_isrForCrosstalkSources_855235_9.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/75d8db9e-a409-42ec-a678-d4cebfb35698_isrForCrosstalkSources_855235_9.stderr parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 4dba55b4f8a3f05d6a6e221ae17f2a55 parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/209025bf-3dc1-43fb-8e22-c21856208635_isrForCrosstalkSources_855241_2.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/209025bf-3dc1-43fb-8e22-c21856208635_isrForCrosstalkSources_855241_2.stderr parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 90885078dba5156c53e17e878aaadbdf parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/55b552b6-be26-458e-96cc-a6ac1cdef193_isrForCrosstalkSources_855231_29.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/55b552b6-be26-458e-96cc-a6ac1cdef193_isrForCrosstalkSources_855231_29.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash fbfe5250204227bfe09a569d50a4e3e6 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/854c85c2-aba8-47ef-8e09-24395831b681_isrForCrosstalkSources_855155_41.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/854c85c2-aba8-47ef-8e09-24395831b681_isrForCrosstalkSources_855155_41.stderr parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 29d95e02c8c2cd5b8eed1d9e76b03cb3 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a6c085c6-41b7-4fbc-8088-4d90735e5814_isrForCrosstalkSources_855242_60.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a6c085c6-41b7-4fbc-8088-4d90735e5814_isrForCrosstalkSources_855242_60.stderr parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash fcfda2bc8a4dc763e38d1d7ad1d6e4b4 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/703936dd-2629-4a56-a95c-7060d5725b52_isrForCrosstalkSources_855236_16.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/703936dd-2629-4a56-a95c-7060d5725b52_isrForCrosstalkSources_855236_16.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 630f5d4661ac5c6616b996fcfbff2e05 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d0399232-073b-40ad-a8c1-937ec1e59082_isrForCrosstalkSources_855233_9.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d0399232-073b-40ad-a8c1-937ec1e59082_isrForCrosstalkSources_855233_9.stderr parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 5d188027c17af97a5897712a835ed087 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a9a08ce0-6d3e-4ac5-a7df-3dcc0d0397cf_isrForCrosstalkSources_855238_51.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a9a08ce0-6d3e-4ac5-a7df-3dcc0d0397cf_isrForCrosstalkSources_855238_51.stderr parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash d384d83008cbe2fffef8062b521e6b76 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/debf286a-95a1-421a-95f1-73520f3c4d23_isrForCrosstalkSources_855231_11.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/debf286a-95a1-421a-95f1-73520f3c4d23_isrForCrosstalkSources_855231_11.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 6d000656949e869fa66781d4a74310a5 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/ffa58946-5b37-4ecd-bc66-ec8449149783_isrForCrosstalkSources_855241_62.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/ffa58946-5b37-4ecd-bc66-ec8449149783_isrForCrosstalkSources_855241_62.stderr parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 98f5b9328c772df5e452b21ed6bd927e parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/29d0666a-78c3-4559-ac88-cd27e80cb359_isrForCrosstalkSources_855234_44.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/29d0666a-78c3-4559-ac88-cd27e80cb359_isrForCrosstalkSources_855234_44.stderr parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 49cb848265723b6923082d846799c715 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4aa88803-8a03-4cb1-a7d4-88120eaf2093_isrForCrosstalkSources_855238_24.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4aa88803-8a03-4cb1-a7d4-88120eaf2093_isrForCrosstalkSources_855238_24.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 8acc02fec6bcc5aefd96991362c5f2b4 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24124ca8-5fbe-4242-b4af-37fc2ecb2414_isrForCrosstalkSources_855241_28.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24124ca8-5fbe-4242-b4af-37fc2ecb2414_isrForCrosstalkSources_855241_28.stderr parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 07ae3a43fe21a1cf6c553f8d75e15c21 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a1837d3f-0886-40bc-a9c6-2d693d4be828_isrForCrosstalkSources_855155_42.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a1837d3f-0886-40bc-a9c6-2d693d4be828_isrForCrosstalkSources_855155_42.stderr parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash caf7844de34d2ba1477677434a7ae767 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/83c5ebd4-ec73-4be1-b4ba-b02a36b03f08_isrForCrosstalkSources_855239_7.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/83c5ebd4-ec73-4be1-b4ba-b02a36b03f08_isrForCrosstalkSources_855239_7.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 9606b937433b3f2617797f46c7c67f15 parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/026bf068-3a25-48d1-8632-d8ff147447ad_isrForCrosstalkSources_855156_51.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/026bf068-3a25-48d1-8632-d8ff147447ad_isrForCrosstalkSources_855156_51.stderr parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 5bb1fa39905323173f086340fff34b2e parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1248e1e8-e028-4890-b591-e3f8a64581b1_isrForCrosstalkSources_855240_2.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1248e1e8-e028-4890-b591-e3f8a64581b1_isrForCrosstalkSources_855240_2.stderr parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 48b31f885f5fc61bf57fd4c7278f5e3f parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a5223054-fdf3-43ea-9ea4-c5cf8dcb06be_isrForCrosstalkSources_855235_33.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a5223054-fdf3-43ea-9ea4-c5cf8dcb06be_isrForCrosstalkSources_855235_33.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 5cbb4b7d8f0bd3a5f2cdd7327809983c parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/4b6650c3-ded7-4fca-9656-31576ef427d7_isrForCrosstalkSources_855235_54.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/4b6650c3-ded7-4fca-9656-31576ef427d7_isrForCrosstalkSources_855235_54.stderr parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 85ab0548c629cfc192c86277c01bfb67 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/c965bd3a-e8bf-44d4-8c57-4ba34e111f69_isrForCrosstalkSources_855233_10.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/c965bd3a-e8bf-44d4-8c57-4ba34e111f69_isrForCrosstalkSources_855233_10.stderr parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 1ce68ca3e3e70c2381b5b3bd263693f5 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5f9fd587-8b3e-468d-b7b2-ed5d04d439bc_isrForCrosstalkSources_855156_62.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5f9fd587-8b3e-468d-b7b2-ed5d04d439bc_isrForCrosstalkSources_855156_62.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 59001f472c48f71c5ca20af96535baed parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/3ed318ba-ff26-48b7-8937-e45cc87be076_isrForCrosstalkSources_855238_29.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/3ed318ba-ff26-48b7-8937-e45cc87be076_isrForCrosstalkSources_855238_29.stderr parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 35d05977613031e14dda27c635def449 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/93a87c44-218b-4fd3-858b-224955072785_isrForCrosstalkSources_855240_34.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/93a87c44-218b-4fd3-858b-224955072785_isrForCrosstalkSources_855240_34.stderr parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash b671d6aa66e6acbcd88b006e079d63d3 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/def94df1-5dfe-44f5-9715-b2030e7b9e67_isrForCrosstalkSources_855234_47.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/def94df1-5dfe-44f5-9715-b2030e7b9e67_isrForCrosstalkSources_855234_47.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash f19e4581d9c0180b42dda3a4639b849e parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4f73749d-ad1f-45e3-8191-b034a6b72727_isrForCrosstalkSources_855156_48.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4f73749d-ad1f-45e3-8191-b034a6b72727_isrForCrosstalkSources_855156_48.stderr parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash c771ee8ec84ce749e3b6b62642fbdf05 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/461d0cf0-f204-46c8-9280-e2bf59f61002_isrForCrosstalkSources_855238_45.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/461d0cf0-f204-46c8-9280-e2bf59f61002_isrForCrosstalkSources_855238_45.stderr parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash 53f5b32a13c64352fe6c5dc7cb49bddf parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/88e013fe-ff19-4761-bb8d-ae1334ec4329_isrForCrosstalkSources_855234_18.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/88e013fe-ff19-4761-bb8d-ae1334ec4329_isrForCrosstalkSources_855234_18.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 47b93166b575ebb52e170f6b50da095f parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4eeaa554-5e30-4f02-ae83-611ba8c0462f_isrForCrosstalkSources_855236_12.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4eeaa554-5e30-4f02-ae83-611ba8c0462f_isrForCrosstalkSources_855236_12.stderr parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 68414eb2b170c9597a9a1b39ad380e7f parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1d5aed23-69c8-4662-ad57-17605999ab3c_isrForCrosstalkSources_855232_42.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1d5aed23-69c8-4662-ad57-17605999ab3c_isrForCrosstalkSources_855232_42.stderr parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash e74ee1a3d2b4b61b74a49a3114f9b443 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/60639732-21a2-4e21-91b0-8a7274826537_isrForCrosstalkSources_855239_9.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/60639732-21a2-4e21-91b0-8a7274826537_isrForCrosstalkSources_855239_9.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash aef2dbfa4642218f657560b41416b201 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914616 proc_lsst.multi INFO: [multi] job_id 16914616 2024-03-08 10:00:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 10:00:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914616 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914616 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16914616 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-08 10:00:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-08 10:00:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 10:00:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6d909492-d87d-4c21-98cc-32a0c311ad30_isrForCrosstalkSources_855240_22.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6d909492-d87d-4c21-98cc-32a0c311ad30_isrForCrosstalkSources_855240_22.stderr parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 542a7dfc636f5bd9a48e27de6adf716e parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0561a62f-9cf9-409f-8dd3-26fb6d9ba15e_isrForCrosstalkSources_855239_56.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0561a62f-9cf9-409f-8dd3-26fb6d9ba15e_isrForCrosstalkSources_855239_56.stderr parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 5e3ae630e94b68ca9ee7170cd3fbe938 parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/480756e3-dbb9-49f2-b651-924067d52eb4_isrForCrosstalkSources_855155_13.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/480756e3-dbb9-49f2-b651-924067d52eb4_isrForCrosstalkSources_855155_13.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash d1b476e7bc314cc36b9fd65704fe4d07 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/42c24ab8-7224-45d5-af73-3f1630801fa3_isrForCrosstalkSources_855154_57.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/42c24ab8-7224-45d5-af73-3f1630801fa3_isrForCrosstalkSources_855154_57.stderr parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 5d4d63db105aa73602f776385704c4d6 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6332ded2-2d91-436a-bde8-b451b7d962f5_isrForCrosstalkSources_855154_53.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6332ded2-2d91-436a-bde8-b451b7d962f5_isrForCrosstalkSources_855154_53.stderr parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash f1e52148b84e799818f0d0df378be3da parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdba840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8ce7b13f-2df9-4477-9acb-d38eab88ba83_isrForCrosstalkSources_855236_6.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8ce7b13f-2df9-4477-9acb-d38eab88ba83_isrForCrosstalkSources_855236_6.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 8e86d59426512634ef82f908d06cc6ac parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6b57054f-9fa3-439d-b03b-11b557b7895c_isrForCrosstalkSources_855240_1.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6b57054f-9fa3-439d-b03b-11b557b7895c_isrForCrosstalkSources_855240_1.stderr parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 01ceecde77030f90f9b367bdfecfdfd2 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f7f5c047-b4e1-4d98-b40a-9f3880135964_isrForCrosstalkSources_855239_48.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f7f5c047-b4e1-4d98-b40a-9f3880135964_isrForCrosstalkSources_855239_48.stderr parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 2b1ccaaef2639af44ddbec513f2ef280 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc532e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/53f5699c-7228-47a0-a8bd-6765efa55150_isrForCrosstalkSources_855241_34.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/53f5699c-7228-47a0-a8bd-6765efa55150_isrForCrosstalkSources_855241_34.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash adcb682da7c11db8baebe91455f4e844 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a6e79388-9251-4245-9fde-106009e3d415_isrForCrosstalkSources_855239_60.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a6e79388-9251-4245-9fde-106009e3d415_isrForCrosstalkSources_855239_60.stderr parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 166a1d6ad0825bd74b648d0c393afc12 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d9b35254-3e51-4f5a-ad9d-ec8c6d26bb62_isrForCrosstalkSources_855232_29.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d9b35254-3e51-4f5a-ad9d-ec8c6d26bb62_isrForCrosstalkSources_855232_29.stderr parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash e7b6e75bd4550d425748c6c440183077 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/379a8651-8fcb-49a9-9d94-99fa4a24d115_isrForCrosstalkSources_855231_34.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/379a8651-8fcb-49a9-9d94-99fa4a24d115_isrForCrosstalkSources_855231_34.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 8efe90bef540b07a3dcfcffc9f342a73 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdb9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a00e7ff9-7c9c-4918-82ec-e84a1116fd87_isrForCrosstalkSources_855156_38.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a00e7ff9-7c9c-4918-82ec-e84a1116fd87_isrForCrosstalkSources_855156_38.stderr parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash ed8c0208e4e7f5103199e86c1191e8b1 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab7cd732-6448-45de-954c-a9db4fd17f57_isrForCrosstalkSources_855235_28.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab7cd732-6448-45de-954c-a9db4fd17f57_isrForCrosstalkSources_855235_28.stderr parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 9a1eb4d7536abdfa282af056db0f6048 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b2111997-2428-402a-a7e7-da518fc904f8_isrForCrosstalkSources_855154_44.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b2111997-2428-402a-a7e7-da518fc904f8_isrForCrosstalkSources_855154_44.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 6d0856dd3d05336e67ca42530d4b8324 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc531a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3117b793-d6d6-4a24-809d-1f6b158f1d49_isrForCrosstalkSources_855241_49.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3117b793-d6d6-4a24-809d-1f6b158f1d49_isrForCrosstalkSources_855241_49.stderr parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash c37a0c3cdddd278a5e81e069867fab3d parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc534c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a5bb0e98-453d-4da5-b10b-dd4c6df91125_isrForCrosstalkSources_855233_2.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a5bb0e98-453d-4da5-b10b-dd4c6df91125_isrForCrosstalkSources_855233_2.stderr parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 197626dcf4a635fd87f94068eb0af441 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d1c63e6a-2c11-4aa1-8f4c-46dc351da93b_isrForCrosstalkSources_855236_5.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d1c63e6a-2c11-4aa1-8f4c-46dc351da93b_isrForCrosstalkSources_855236_5.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash ed34a1307fba576617787ad661196c9c parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2366332e-dda8-4328-a2e8-9af2931a1530_isrForCrosstalkSources_855242_15.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2366332e-dda8-4328-a2e8-9af2931a1530_isrForCrosstalkSources_855242_15.stderr parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash aeb1c73420d8d01ea57c1919aee11310 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b369ea40-cc33-4d0c-91b1-693363ec2ded_isrForCrosstalkSources_855233_19.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b369ea40-cc33-4d0c-91b1-693363ec2ded_isrForCrosstalkSources_855233_19.stderr parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash 76aa643a7be46996ac95820b91648f11 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc500e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/bbacf5ae-b941-4954-9068-a0a74d3a33ac_isrForCrosstalkSources_855235_50.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/bbacf5ae-b941-4954-9068-a0a74d3a33ac_isrForCrosstalkSources_855235_50.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash a6ca901c944ebcff26fe6d4a234021dc parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7b79bb10-cf4f-4662-bc33-0d6358504b46_isrForCrosstalkSources_855241_60.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7b79bb10-cf4f-4662-bc33-0d6358504b46_isrForCrosstalkSources_855241_60.stderr parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 86927273d01dc7f49b37965f4424bd88 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/4712b25d-5bfe-427d-b5d8-60486207ca84_isrForCrosstalkSources_855155_20.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/4712b25d-5bfe-427d-b5d8-60486207ca84_isrForCrosstalkSources_855155_20.stderr parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 79506158b102ab061060d32f70628a52 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2bcde293-2936-4903-b36d-98569584065c_isrForCrosstalkSources_855239_14.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2bcde293-2936-4903-b36d-98569584065c_isrForCrosstalkSources_855239_14.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 206f1a68aa65cffb741c54afa6fdf7a0 parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/442ede9c-5a4f-4657-b899-3e61cbd833b9_isrForCrosstalkSources_855236_25.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/442ede9c-5a4f-4657-b899-3e61cbd833b9_isrForCrosstalkSources_855236_25.stderr parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 1dbd2e557967743619a453c44c697789 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/77be2220-660c-42e3-9e98-580fac7c4f4d_isrForCrosstalkSources_855234_42.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/77be2220-660c-42e3-9e98-580fac7c4f4d_isrForCrosstalkSources_855234_42.stderr parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash 9094a7c85c835dcb752d781a7efb07a0 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc502c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/35cc9212-e2d4-472f-9391-c76151e3f4ff_isrForCrosstalkSources_855154_21.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/35cc9212-e2d4-472f-9391-c76151e3f4ff_isrForCrosstalkSources_855154_21.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 9f7d8cdae87c4f84bfc848df15a98083 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1486ffbd-7fb0-4182-896c-ec3e256ec271_isrForCrosstalkSources_855237_18.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1486ffbd-7fb0-4182-896c-ec3e256ec271_isrForCrosstalkSources_855237_18.stderr parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash b531d11fd68aa5749d21eb5880fc87ed parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc504a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914617 proc_lsst.multi INFO: [multi] job_id 16914617 2024-03-08 10:00:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 10:00:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914617 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914617 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16914617 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-08 10:00:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-08 10:00:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 10:00:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/4915eef4-b636-4b50-946a-2e5fd23e23f5_isrForCrosstalkSources_855239_12.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/4915eef4-b636-4b50-946a-2e5fd23e23f5_isrForCrosstalkSources_855239_12.stderr parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash cb77baf69141525386febc1940fe5a74 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/dadadd3c-a810-414c-830e-7778e9a8bbf2_isrForCrosstalkSources_855239_53.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/dadadd3c-a810-414c-830e-7778e9a8bbf2_isrForCrosstalkSources_855239_53.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash bee9f4389a1a5f8342196fbc562b7fd2 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f919ea81-1511-4c87-b554-7dee80903211_isrForCrosstalkSources_855232_54.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f919ea81-1511-4c87-b554-7dee80903211_isrForCrosstalkSources_855232_54.stderr parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash fb481d9ec03df739a5e5e7b00bebc893 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c2b06ab3-e5b0-430a-946e-49e2f8c48e0c_isrForCrosstalkSources_855241_61.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c2b06ab3-e5b0-430a-946e-49e2f8c48e0c_isrForCrosstalkSources_855241_61.stderr parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 06d97087df8221872a166c2371f13dc3 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc507c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/22a24431-1174-47cd-a24c-bade86d0c01c_isrForCrosstalkSources_855242_16.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/22a24431-1174-47cd-a24c-bade86d0c01c_isrForCrosstalkSources_855242_16.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 0459c5fe9f401a1046952ed172e1dab2 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7c4aceb8-ec33-462c-989c-2d57c4b3b515_isrForCrosstalkSources_855232_28.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7c4aceb8-ec33-462c-989c-2d57c4b3b515_isrForCrosstalkSources_855232_28.stderr parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 870a81f0e76ea6f1c432a336e55a2cbd parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1554753b-5fed-47d3-8a26-f025b1f2cff9_isrForCrosstalkSources_855237_62.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1554753b-5fed-47d3-8a26-f025b1f2cff9_isrForCrosstalkSources_855237_62.stderr parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 02b95ea5f06f1abe1d11134df79b95e2 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/54014cfd-1d6b-4320-a8ff-f40eae9d8582_isrForCrosstalkSources_855231_62.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/54014cfd-1d6b-4320-a8ff-f40eae9d8582_isrForCrosstalkSources_855231_62.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash 9c7e584adb579e7ff2097938d37f5321 parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/38e40d28-5d4d-4c38-8c9e-e05077677cce_isrForCrosstalkSources_855233_33.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/38e40d28-5d4d-4c38-8c9e-e05077677cce_isrForCrosstalkSources_855233_33.stderr parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 68e4023ddcae45206f9af5a07d556f41 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc509a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f9c2d9b4-9d24-4d33-8777-a6ecfb1464c7_isrForCrosstalkSources_855242_23.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f9c2d9b4-9d24-4d33-8777-a6ecfb1464c7_isrForCrosstalkSources_855242_23.stderr parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash bf90fb73cf5cae471f9db134f40da217 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b0eb0223-d739-48a2-adad-933ea204c962_isrForCrosstalkSources_855242_44.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b0eb0223-d739-48a2-adad-933ea204c962_isrForCrosstalkSources_855242_44.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 236e0a02a1c55215d6729000a2e43d58 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/668ef56d-c868-4ded-857d-d227669b071f_isrForCrosstalkSources_855239_54.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/668ef56d-c868-4ded-857d-d227669b071f_isrForCrosstalkSources_855239_54.stderr parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash f46b3532940e206d460fc04c1bb8e167 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e78992a9-1501-4e1b-abdc-b5f55474758c_isrForCrosstalkSources_855232_38.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e78992a9-1501-4e1b-abdc-b5f55474758c_isrForCrosstalkSources_855232_38.stderr parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash bd2abca384fbe02d4c816a796ed08425 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/24aa56f1-a5e9-438e-a679-b3811e075228_isrForCrosstalkSources_855156_47.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/24aa56f1-a5e9-438e-a679-b3811e075228_isrForCrosstalkSources_855156_47.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 31efbf0e41db70092ae073e7cd01b6a8 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0811f197-6794-47fc-82d7-be25d980dec1_isrForCrosstalkSources_855241_48.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0811f197-6794-47fc-82d7-be25d980dec1_isrForCrosstalkSources_855241_48.stderr parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 02614a4049ce14414cdcd7d4617e36f7 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d2c92434-4d94-440f-b530-1c6ad5595df8_isrForCrosstalkSources_855156_19.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d2c92434-4d94-440f-b530-1c6ad5595df8_isrForCrosstalkSources_855156_19.stderr parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 671975d346a45ca0e0574976921afa83 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/43cba040-6952-44b5-accb-d6adc1abe8ed_isrForCrosstalkSources_855231_18.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/43cba040-6952-44b5-accb-d6adc1abe8ed_isrForCrosstalkSources_855231_18.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 6dc60906b8adc51c11dd14c3be19d264 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/192b3f35-f64f-46f2-b809-f00572758f12_isrForCrosstalkSources_855156_12.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/192b3f35-f64f-46f2-b809-f00572758f12_isrForCrosstalkSources_855156_12.stderr parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash 846a0da31fef11404646ebcdd4d43a7a parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0d31442e-09d1-4747-ade7-21d3ece07c70_isrForCrosstalkSources_855231_40.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0d31442e-09d1-4747-ade7-21d3ece07c70_isrForCrosstalkSources_855231_40.stderr parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 161406d3e0059f6572a0f23938a09767 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f9a25a96-644d-46f5-9c67-fe1c08035ec8_isrForCrosstalkSources_855236_44.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f9a25a96-644d-46f5-9c67-fe1c08035ec8_isrForCrosstalkSources_855236_44.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 5ebb4c8a06bf91d9873584756e884c80 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/12d3d6a9-1101-4886-935f-7b376a5bda7c_isrForCrosstalkSources_855156_57.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/12d3d6a9-1101-4886-935f-7b376a5bda7c_isrForCrosstalkSources_855156_57.stderr parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash de9c6795ce5b249e328d49eed631c121 parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4409821b-c831-48fe-891e-0d926c3756b3_isrForCrosstalkSources_855237_27.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4409821b-c831-48fe-891e-0d926c3756b3_isrForCrosstalkSources_855237_27.stderr parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 0efa58a4a15864bff1bebe7b3559c9ab parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/80a5f7b4-b449-43e2-92c1-4764acde3785_isrForCrosstalkSources_855233_40.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/80a5f7b4-b449-43e2-92c1-4764acde3785_isrForCrosstalkSources_855233_40.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 2bf18187fcd368989505dad78e2d544b parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/dc6ce576-cc94-4fdf-bb12-855377810786_isrForCrosstalkSources_855232_45.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/dc6ce576-cc94-4fdf-bb12-855377810786_isrForCrosstalkSources_855232_45.stderr parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 90e2f6a56f2ab13ab98e868fa4dc7515 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc539c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f89a2aa2-5632-4da6-9de6-c9ba4b098148_isrForCrosstalkSources_855234_13.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f89a2aa2-5632-4da6-9de6-c9ba4b098148_isrForCrosstalkSources_855234_13.stderr parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 4d5c07dc463b85fc6c567ce4315ae050 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ae0d54e2-e03e-486a-8dc8-228f8b8a4b4a_isrForCrosstalkSources_855238_8.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ae0d54e2-e03e-486a-8dc8-228f8b8a4b4a_isrForCrosstalkSources_855238_8.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 4c363a2a8fff74d5a5f3d06ee8619224 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/068f5736-edb2-4a50-bcea-19ef18a08c4a_isrForCrosstalkSources_855241_26.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/068f5736-edb2-4a50-bcea-19ef18a08c4a_isrForCrosstalkSources_855241_26.stderr parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 66876d69aae3615405a76bf76d1bd58b parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7140d445-cc90-43eb-955f-443f214eef5a_isrForCrosstalkSources_855237_53.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7140d445-cc90-43eb-955f-443f214eef5a_isrForCrosstalkSources_855237_53.stderr parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 5fd20e5207ba75e34f086d3135cf82f9 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72bea11b-04e8-4a21-8131-43b53d03bdaa_isrForCrosstalkSources_855240_47.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72bea11b-04e8-4a21-8131-43b53d03bdaa_isrForCrosstalkSources_855240_47.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 09b6d0a828859e28d25e3d3b2f5083c7 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4d9edb75-7373-44ff-9020-81da6233d6c7_isrForCrosstalkSources_855242_42.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4d9edb75-7373-44ff-9020-81da6233d6c7_isrForCrosstalkSources_855242_42.stderr parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 80f88fbe9d0ee2ca0fef28a555aaadb3 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fafd2f53-e79d-4935-95d8-fe0e43235cec_isrForCrosstalkSources_855239_50.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fafd2f53-e79d-4935-95d8-fe0e43235cec_isrForCrosstalkSources_855239_50.stderr parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 1890d21a3a3c96d86a813e51af5e94b9 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d305ce3-a7c3-4151-805c-bf877b40dc23_isrForCrosstalkSources_855237_60.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d305ce3-a7c3-4151-805c-bf877b40dc23_isrForCrosstalkSources_855237_60.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 867d5aad68006382bd1d8300adc919c7 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/81447dcb-6b6f-49f3-847c-d0cf2a842720_isrForCrosstalkSources_855233_21.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/81447dcb-6b6f-49f3-847c-d0cf2a842720_isrForCrosstalkSources_855233_21.stderr parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash daf5a4c345dbf0f76989ea4a84398744 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/93aa32b2-f04b-4bd5-8863-410fd5e2e80d_isrForCrosstalkSources_855236_9.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/93aa32b2-f04b-4bd5-8863-410fd5e2e80d_isrForCrosstalkSources_855236_9.stderr parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 85cbd754b6296099cf19b12896f33316 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dac00e95-6222-4ec9-bad9-5962ca5c585b_isrForCrosstalkSources_855154_40.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dac00e95-6222-4ec9-bad9-5962ca5c585b_isrForCrosstalkSources_855154_40.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 250544eb245391ebe3d01bb4039ae4ec parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6e1fa75b-4635-40a2-aa52-a756d4e461af_isrForCrosstalkSources_855235_19.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6e1fa75b-4635-40a2-aa52-a756d4e461af_isrForCrosstalkSources_855235_19.stderr parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash af3c5519d30912270bba09e69bfa4f47 parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e29b3d28-f0dd-47a1-b0a5-c596402b1199_isrForCrosstalkSources_855242_22.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e29b3d28-f0dd-47a1-b0a5-c596402b1199_isrForCrosstalkSources_855242_22.stderr parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 70d7caa53a8f2bdaf38f6bbf67bc44b4 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/16e17670-6667-49d0-b281-6bd2e9619054_isrForCrosstalkSources_855237_6.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/16e17670-6667-49d0-b281-6bd2e9619054_isrForCrosstalkSources_855237_6.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 3fe2f84949b63f7bcebdc230e421d99a parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914618 proc_lsst.multi INFO: [multi] job_id 16914618 2024-03-08 10:00:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 10:00:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914618 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914618 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16914618 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-08 10:00:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-08 10:00:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 10:00:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/eb6f3009-be63-4af1-9fe5-f27ab7ed2f1b_isrForCrosstalkSources_855241_21.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/eb6f3009-be63-4af1-9fe5-f27ab7ed2f1b_isrForCrosstalkSources_855241_21.stderr parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash e2db9b9a1df8093b67a3027fc6bfc51b parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aab6d2b4-adf8-4818-83df-f8530fba4df6_isrForCrosstalkSources_855242_28.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aab6d2b4-adf8-4818-83df-f8530fba4df6_isrForCrosstalkSources_855242_28.stderr parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 1ec4a36d5663cb631b4ae494bb7eb604 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc50e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/829c9f74-2ecf-43f3-b12a-5ef50bc51f26_isrForCrosstalkSources_855236_40.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/829c9f74-2ecf-43f3-b12a-5ef50bc51f26_isrForCrosstalkSources_855236_40.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash d6479e2c606d0b36cb2c1c40e3ecf885 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0cf84850-5a63-404d-9368-5bb29221a472_isrForCrosstalkSources_855155_57.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0cf84850-5a63-404d-9368-5bb29221a472_isrForCrosstalkSources_855155_57.stderr parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 973bfc853a697255f4382b6752a3d62c parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0ea5beef-5579-405c-b01b-29471a2f8bfb_isrForCrosstalkSources_855239_18.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0ea5beef-5579-405c-b01b-29471a2f8bfb_isrForCrosstalkSources_855239_18.stderr parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 813c496c5f17ed30cbb53056c287c941 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc518a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/981a4276-8423-43f3-a860-18b67565df26_isrForCrosstalkSources_855232_5.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/981a4276-8423-43f3-a860-18b67565df26_isrForCrosstalkSources_855232_5.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 009ac13a0b7386d9f57f1e67706c7883 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc516c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5e01925f-0ffb-4a9b-8139-f6937cb961f7_isrForCrosstalkSources_855156_13.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5e01925f-0ffb-4a9b-8139-f6937cb961f7_isrForCrosstalkSources_855156_13.stderr parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash aa4d8b7cf0ab44d130c5043f2fc6ba84 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5f8751f4-7f3b-4132-bac7-713cb54eed40_isrForCrosstalkSources_855242_54.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5f8751f4-7f3b-4132-bac7-713cb54eed40_isrForCrosstalkSources_855242_54.stderr parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash cd1c0fa9fa78b845ac2248505ba284f7 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/62efd87c-f514-408a-ad63-df779d7c7a81_isrForCrosstalkSources_855238_13.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/62efd87c-f514-408a-ad63-df779d7c7a81_isrForCrosstalkSources_855238_13.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash ec4cf43fa753d8a0efdb47437827e111 parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/02abc455-ffcf-433f-9dda-4fd758f73cf2_isrForCrosstalkSources_855239_39.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/02abc455-ffcf-433f-9dda-4fd758f73cf2_isrForCrosstalkSources_855239_39.stderr parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 5f64423b9c8b686c8c2ee44ed0863c1d parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/44304a09-107b-4ab6-ae25-dd26bbbf392f_isrForCrosstalkSources_855232_40.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/44304a09-107b-4ab6-ae25-dd26bbbf392f_isrForCrosstalkSources_855232_40.stderr parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 2400fe93e3e5ca2c87d79e8646c300ad parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8a79515b-6e1b-4317-9204-5611b578b618_isrForCrosstalkSources_855154_35.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8a79515b-6e1b-4317-9204-5611b578b618_isrForCrosstalkSources_855154_35.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 4639f578fed7eb562dded4dd48e993a1 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/baa03f1f-9b81-4c6d-ba8a-b5e85b53535c_isrForCrosstalkSources_855155_31.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/baa03f1f-9b81-4c6d-ba8a-b5e85b53535c_isrForCrosstalkSources_855155_31.stderr parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 69a5c31f8e25cce6927b1dee44f4c206 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/57622a83-55f7-43a7-8e2a-ff304e4291aa_isrForCrosstalkSources_855234_56.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/57622a83-55f7-43a7-8e2a-ff304e4291aa_isrForCrosstalkSources_855234_56.stderr parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash f4bc95a81c938565e599206342839d03 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1dc92e6d-0db8-4e87-b6aa-2a6b180402b9_isrForCrosstalkSources_855237_55.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1dc92e6d-0db8-4e87-b6aa-2a6b180402b9_isrForCrosstalkSources_855237_55.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash e5b95d9afba01c09f82be91eb340ff07 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d36b92eb-ea89-4946-9aa0-d3f772000c95_isrForCrosstalkSources_855155_11.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d36b92eb-ea89-4946-9aa0-d3f772000c95_isrForCrosstalkSources_855155_11.stderr parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash dfafd16f44ce426d97c8bad7356a8156 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc527a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/5cd183af-8a9c-4817-9784-601c6a73e62d_isrForCrosstalkSources_855241_58.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/5cd183af-8a9c-4817-9784-601c6a73e62d_isrForCrosstalkSources_855241_58.stderr parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash fc9fac9c099be30c3cd01253d350f185 parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc525c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1d3d9fe2-5d8d-486d-81c6-de31d15e938d_isrForCrosstalkSources_855239_8.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1d3d9fe2-5d8d-486d-81c6-de31d15e938d_isrForCrosstalkSources_855239_8.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 2609c6fa95cd47dda45d8455eb26656a parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d89d3e97-5202-438b-bbe9-e6628fff7a6b_isrForCrosstalkSources_855234_50.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d89d3e97-5202-438b-bbe9-e6628fff7a6b_isrForCrosstalkSources_855234_50.stderr parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash b4fa161e96a8f4125c6572acd6af439b parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c75aaacf-75e7-4b4c-a687-d371476b2c71_isrForCrosstalkSources_855240_26.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c75aaacf-75e7-4b4c-a687-d371476b2c71_isrForCrosstalkSources_855240_26.stderr parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 24d91ce99dffcfbe4a3855e32e46d8e9 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d6ff70fa-bd01-44a4-b20b-bd85da6619b1_isrForCrosstalkSources_855238_6.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d6ff70fa-bd01-44a4-b20b-bd85da6619b1_isrForCrosstalkSources_855238_6.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 7593fdc2341b0dc5ed480ce6ade0f200 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2077b9f7-9fdc-4ec0-9162-6e7a1248d901_isrForCrosstalkSources_855154_10.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2077b9f7-9fdc-4ec0-9162-6e7a1248d901_isrForCrosstalkSources_855154_10.stderr parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash e5c37802a135d6d35401e1289f3dfd49 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6317b03d-26c5-4b81-9f2a-eee46aa422e9_isrForCrosstalkSources_855242_52.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6317b03d-26c5-4b81-9f2a-eee46aa422e9_isrForCrosstalkSources_855242_52.stderr parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash a40f138286f06e1b115149f81264e0ee parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc523e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ed777137-f810-42e5-af84-3fe8944cb7a7_isrForCrosstalkSources_855234_23.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ed777137-f810-42e5-af84-3fe8944cb7a7_isrForCrosstalkSources_855234_23.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 7d9f53159d59ffd2d0ec79c16f200a82 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/499516d9-bbb0-4d90-b27c-996bb24eb277_isrForCrosstalkSources_855231_8.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/499516d9-bbb0-4d90-b27c-996bb24eb277_isrForCrosstalkSources_855231_8.stderr parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 69667697c0d1e64fe6516e398f656b38 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc520c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bbd6520c-b81e-446c-9df8-2c4997c1b37b_isrForCrosstalkSources_855155_53.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bbd6520c-b81e-446c-9df8-2c4997c1b37b_isrForCrosstalkSources_855155_53.stderr parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 20cc64802447aa72881e9f7143b4761e parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc522a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2fdd689a-e4f8-40ad-a2de-23461ca1b99c_isrForCrosstalkSources_855239_26.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2fdd689a-e4f8-40ad-a2de-23461ca1b99c_isrForCrosstalkSources_855239_26.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 0f0ff11d893c13fc236244e386ffe1f6 parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc536a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a545c06f-16e3-4756-8385-23e99e9be439_isrForCrosstalkSources_855233_34.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a545c06f-16e3-4756-8385-23e99e9be439_isrForCrosstalkSources_855233_34.stderr parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash f855813c5816600577f2c43d9dad6eed parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4d2b5265-9ccb-4e8c-9041-5e1ad68752a0_isrForCrosstalkSources_855231_37.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4d2b5265-9ccb-4e8c-9041-5e1ad68752a0_isrForCrosstalkSources_855231_37.stderr parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash bb92cb98bfe1c5a3255a85b4746144fd parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b24360ff-fded-4866-8d76-99a9dad103ba_isrForCrosstalkSources_855231_27.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b24360ff-fded-4866-8d76-99a9dad103ba_isrForCrosstalkSources_855231_27.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 4e60839c390c6dde77ac386361c67cda parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e72ccc27-a9f8-4267-bae1-c2772aef2ba5_isrForCrosstalkSources_855235_46.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e72ccc27-a9f8-4267-bae1-c2772aef2ba5_isrForCrosstalkSources_855235_46.stderr parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 07cf7a164ee23eb279f31b58ace425b9 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d8d25cfe-1898-4218-90f9-f410bf825623_isrForCrosstalkSources_855238_19.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d8d25cfe-1898-4218-90f9-f410bf825623_isrForCrosstalkSources_855238_19.stderr parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash cf5e420494e3b2be5b2c30d80ee7f160 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc537e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914619 proc_lsst.multi INFO: [multi] job_id 16914619 2024-03-08 10:00:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:00:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914619 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914619 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16914619 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-08 10:00:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-08 10:00:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:00:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/21d2cf13-f20a-4496-8819-7f10e32b0a65_isrForCrosstalkSources_855238_39.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/21d2cf13-f20a-4496-8819-7f10e32b0a65_isrForCrosstalkSources_855238_39.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash a80f052db270172a20cfa13817c70433 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ae5e31b5-899b-47d4-96db-cd1d4c151853_isrForCrosstalkSources_855237_61.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ae5e31b5-899b-47d4-96db-cd1d4c151853_isrForCrosstalkSources_855237_61.stderr parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 9f40f63d1fee43c84e9e4087c4d64c60 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2995019e-40c9-4ff9-8d18-0ceeed740f52_isrForCrosstalkSources_855234_53.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2995019e-40c9-4ff9-8d18-0ceeed740f52_isrForCrosstalkSources_855234_53.stderr parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash c9205c47af9dd0307e1ec7e414d86237 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/59353850-319f-4614-ab6e-f342406a5e6e_isrForCrosstalkSources_855238_58.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/59353850-319f-4614-ab6e-f342406a5e6e_isrForCrosstalkSources_855238_58.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 5069949af7b529eb8320dc9b2c62e644 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc51440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/dcf4572c-7cf9-47e7-bee7-e0af51a1dfb6_isrForCrosstalkSources_855233_58.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/dcf4572c-7cf9-47e7-bee7-e0af51a1dfb6_isrForCrosstalkSources_855233_58.stderr parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash c148b031e74a7172cd8c5f3bd962b423 parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc528e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c834ab62-0749-445e-ae5b-d65908257c05_isrForCrosstalkSources_855241_42.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c834ab62-0749-445e-ae5b-d65908257c05_isrForCrosstalkSources_855241_42.stderr parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash adfc3ad422adfbe9824488db86da0f5f parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f0d6ee1d-0caf-462b-8030-1f4325547d94_isrForCrosstalkSources_855232_8.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f0d6ee1d-0caf-462b-8030-1f4325547d94_isrForCrosstalkSources_855232_8.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 77a7a7e4331dfb9f4ce5e9796f52fb3c parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc505e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/ec26d339-043f-4eab-9421-4cf628f70196_isrForCrosstalkSources_855156_4.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/ec26d339-043f-4eab-9421-4cf628f70196_isrForCrosstalkSources_855156_4.stderr parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 9c9d6b03f021ce89aec2f83acf902e2e parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc53f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/017308e7-9d76-44a6-95e3-cfbdd32869f8_isrForCrosstalkSources_855241_57.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/017308e7-9d76-44a6-95e3-cfbdd32869f8_isrForCrosstalkSources_855241_57.stderr parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash a6dbb96206b8274966d2365b12d8ca94 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfc52980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/20d03dd8-4222-4fab-b7f8-c047900d59cb_isrForCrosstalkSources_855240_10.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/20d03dd8-4222-4fab-b7f8-c047900d59cb_isrForCrosstalkSources_855240_10.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 9e0f141e7758332cccc27443b56b561c parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7d1b7a04-dbef-4f61-afa5-d10254b0bc00_isrForCrosstalkSources_855155_62.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7d1b7a04-dbef-4f61-afa5-d10254b0bc00_isrForCrosstalkSources_855155_62.stderr parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash 9e6d4d97198430456c106a72fda03324 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b82c7b13-61ac-42b9-94d9-6e2e2438113a_isrForCrosstalkSources_855235_60.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b82c7b13-61ac-42b9-94d9-6e2e2438113a_isrForCrosstalkSources_855235_60.stderr parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash bd19b69710e9590e702043d85ec781d0 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5f3c49e2-e9a8-4db9-b985-0a6b1e4f8952_isrForCrosstalkSources_855238_31.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5f3c49e2-e9a8-4db9-b985-0a6b1e4f8952_isrForCrosstalkSources_855238_31.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 9c8af3a3a869438de857c06816305295 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/80c37bad-8d39-43df-b4a8-d78d099b881c_isrForCrosstalkSources_855237_35.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/80c37bad-8d39-43df-b4a8-d78d099b881c_isrForCrosstalkSources_855237_35.stderr parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash c64ba49c4bdfc6385a0c03f07fde686d parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/de060991-6ef2-49a7-8cc7-9da4888e61c8_isrForCrosstalkSources_855237_15.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/de060991-6ef2-49a7-8cc7-9da4888e61c8_isrForCrosstalkSources_855237_15.stderr parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 4c0ca4bf901ce147d982ae769e65f80c parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/534714b7-e337-4b62-ba01-5c9fb8b31414_isrForCrosstalkSources_855234_27.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/534714b7-e337-4b62-ba01-5c9fb8b31414_isrForCrosstalkSources_855234_27.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 708fdf72dd1fbc3ad2cf9ebca7ccd10d parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/02b08377-dcaa-47c2-8613-33b4d4dd092c_isrForCrosstalkSources_855232_23.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/02b08377-dcaa-47c2-8613-33b4d4dd092c_isrForCrosstalkSources_855232_23.stderr parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash a9cd88a35acca81aafcc8b001b71ca8e parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/edf91535-e1c5-418c-b58e-2ec5dfafb2a0_isrForCrosstalkSources_855154_52.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/edf91535-e1c5-418c-b58e-2ec5dfafb2a0_isrForCrosstalkSources_855154_52.stderr parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 2afced481afde92a9276a4ffc0a37831 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a803e7fa-74db-445e-802f-8a37df69e422_isrForCrosstalkSources_855239_46.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a803e7fa-74db-445e-802f-8a37df69e422_isrForCrosstalkSources_855239_46.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 96b11b0eb1157312d70991e80424be2a parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3aad2da6-4a7e-48d3-ab8e-f47cce2cf095_isrForCrosstalkSources_855155_40.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3aad2da6-4a7e-48d3-ab8e-f47cce2cf095_isrForCrosstalkSources_855155_40.stderr parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 379a91462428bcac1b04e87cfd1471e5 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/15409bfe-7ce6-4727-953f-a1bf77d67604_isrForCrosstalkSources_855233_39.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/15409bfe-7ce6-4727-953f-a1bf77d67604_isrForCrosstalkSources_855233_39.stderr parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 0c1c76aa425bee3eb9d3560c0921d546 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4733ab5b-b83c-4894-aa2d-79ccf1978292_isrForCrosstalkSources_855154_34.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4733ab5b-b83c-4894-aa2d-79ccf1978292_isrForCrosstalkSources_855154_34.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash cef577515b38796ce4c6a1c3ccf62107 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/25a7fcd5-befa-429b-8047-425863b0726d_isrForCrosstalkSources_855236_14.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/25a7fcd5-befa-429b-8047-425863b0726d_isrForCrosstalkSources_855236_14.stderr parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 8757c30101eca3f6f76837fb8e1b1593 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/32366752-1055-4a6f-b908-6986623803ca_isrForCrosstalkSources_855242_30.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/32366752-1055-4a6f-b908-6986623803ca_isrForCrosstalkSources_855242_30.stderr parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash eaa0391407cb951922a0d3ee11ad5ebb parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a8b0576-5c28-4620-9995-8c65054cae46_isrForCrosstalkSources_855235_7.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a8b0576-5c28-4620-9995-8c65054cae46_isrForCrosstalkSources_855235_7.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash c0cfa0a6b1e95dd6ca2aee57c3b1857a parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/bbd12cf1-a484-42b6-92b5-6d8c1ced5173_isrForCrosstalkSources_855231_6.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/bbd12cf1-a484-42b6-92b5-6d8c1ced5173_isrForCrosstalkSources_855231_6.stderr parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 78fdabda0de433c91d57b1dcaa147f8f parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/841d5d28-0864-451e-bfe9-9c6a2d470e40_isrForCrosstalkSources_855233_3.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/841d5d28-0864-451e-bfe9-9c6a2d470e40_isrForCrosstalkSources_855233_3.stderr parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash eaa13219bec94135a2cdd57b017a1c1e parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e0b86132-cbac-47d6-b392-f7592949606b_isrForCrosstalkSources_855237_37.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e0b86132-cbac-47d6-b392-f7592949606b_isrForCrosstalkSources_855237_37.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 0a1c996f1a8d79dd5acaf2b1444fb231 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/03855faf-086d-439e-b680-8424d72f8a62_isrForCrosstalkSources_855233_31.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/03855faf-086d-439e-b680-8424d72f8a62_isrForCrosstalkSources_855233_31.stderr parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 39e37f389dc36d1cb11566114538c66d parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5175a26a-5c7e-49cf-974f-1073b33cc0c4_isrForCrosstalkSources_855233_15.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5175a26a-5c7e-49cf-974f-1073b33cc0c4_isrForCrosstalkSources_855233_15.stderr parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash b94b12ec2a289592da2dd4b3c2e056c9 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3cc28b9-421b-4075-8e72-147149b2eb7e_isrForCrosstalkSources_855236_15.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3cc28b9-421b-4075-8e72-147149b2eb7e_isrForCrosstalkSources_855236_15.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 2ba2387e56ebecf8c576bcb5cef1d151 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/76918276-0d1e-4f1c-8081-d2eacbc4a65d_isrForCrosstalkSources_855231_17.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/76918276-0d1e-4f1c-8081-d2eacbc4a65d_isrForCrosstalkSources_855231_17.stderr parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash c51bec1418bce3b05d611fe63ef7395b parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bb46a161-7a60-4678-a045-0e206f610b6c_isrForCrosstalkSources_855155_22.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bb46a161-7a60-4678-a045-0e206f610b6c_isrForCrosstalkSources_855155_22.stderr parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash e6d4b5508a929f1b0d17e3ee35c1b4f7 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de7aeb26-70c0-42f0-b306-f1a78a1fd7a2_isrForCrosstalkSources_855233_52.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de7aeb26-70c0-42f0-b306-f1a78a1fd7a2_isrForCrosstalkSources_855233_52.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 090e5d43b3a921eaff12720e52efa32b parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c9146d2-97fe-4d9f-b8d9-6882e294b244_isrForCrosstalkSources_855241_54.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c9146d2-97fe-4d9f-b8d9-6882e294b244_isrForCrosstalkSources_855241_54.stderr parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 707b9ed9e7a4d8afe24bf9cce66d90aa parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914620 proc_lsst.multi INFO: [multi] job_id 16914620 2024-03-08 10:00:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 10:00:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914620 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914620 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16914620 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-08 10:00:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-08 10:00:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 10:00:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9639e41f-daa6-40f2-ac21-922ab52fd7a7_isrForCrosstalkSources_855231_61.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9639e41f-daa6-40f2-ac21-922ab52fd7a7_isrForCrosstalkSources_855231_61.stderr parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 53a0e65924fab16c2848afabf9262a75 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/67000f2d-142a-4a17-9c1d-8360bcc11e8e_isrForCrosstalkSources_855237_48.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/67000f2d-142a-4a17-9c1d-8360bcc11e8e_isrForCrosstalkSources_855237_48.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 8344568ad50015cb1ea2e725b0acb4b8 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2419a22d-8abe-4b99-b8bc-96d08abe028f_isrForCrosstalkSources_855239_38.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2419a22d-8abe-4b99-b8bc-96d08abe028f_isrForCrosstalkSources_855239_38.stderr parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 176effe34cfeb08921c1eca234dc1609 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9f86ed45-7f63-4f41-a0d4-9887de11f536_isrForCrosstalkSources_855233_13.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9f86ed45-7f63-4f41-a0d4-9887de11f536_isrForCrosstalkSources_855233_13.stderr parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 532a782346f5124e2f5ee3d4bbc268f1 parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d9c6ab32-43c8-4af8-bc57-0ccdde322770_isrForCrosstalkSources_855234_4.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d9c6ab32-43c8-4af8-bc57-0ccdde322770_isrForCrosstalkSources_855234_4.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 75b6dd29b4e18f857164bcbf4404ac5b parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ccba67e3-9e64-4100-9d00-04625a397736_isrForCrosstalkSources_855155_27.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ccba67e3-9e64-4100-9d00-04625a397736_isrForCrosstalkSources_855155_27.stderr parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 533b3681835eb95f2e38af34fa6cfc2e parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/32eac1bf-5f44-48c9-947f-ba060e785917_isrForCrosstalkSources_855241_56.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/32eac1bf-5f44-48c9-947f-ba060e785917_isrForCrosstalkSources_855241_56.stderr parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 16dc58ad806a008f10bdf09e34b624a2 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/42393a68-e71b-4300-947f-76013505a75f_isrForCrosstalkSources_855241_32.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/42393a68-e71b-4300-947f-76013505a75f_isrForCrosstalkSources_855241_32.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 6acf77cf8007846c3484ba7470b434e2 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c3dd3a57-deda-4c9a-ab31-3ae0ec5ac26f_isrForCrosstalkSources_855156_3.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c3dd3a57-deda-4c9a-ab31-3ae0ec5ac26f_isrForCrosstalkSources_855156_3.stderr parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash b45a2a56bf5df28d04477b250ea46985 parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1e50c119-9a73-4a0d-9c8b-4619980b9f02_isrForCrosstalkSources_855239_30.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1e50c119-9a73-4a0d-9c8b-4619980b9f02_isrForCrosstalkSources_855239_30.stderr parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 718f47968e91259911ce2f4a025de99f parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/8c77cd27-ddbf-4f48-9071-de3b70c86d0e_isrForCrosstalkSources_855242_36.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/8c77cd27-ddbf-4f48-9071-de3b70c86d0e_isrForCrosstalkSources_855242_36.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 3b2c712fd66ded6397d0ab5a9ead27f8 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a9c36e0f-a88e-492b-a179-5d226d6b1df4_isrForCrosstalkSources_855234_30.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a9c36e0f-a88e-492b-a179-5d226d6b1df4_isrForCrosstalkSources_855234_30.stderr parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash faf966ef623a27235e9464d8d30c044d parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/22b60404-cba1-48e2-9218-24061b3d0868_isrForCrosstalkSources_855156_14.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/22b60404-cba1-48e2-9218-24061b3d0868_isrForCrosstalkSources_855156_14.stderr parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash e6cd1a9d8330b8df1e1f2cd7416f6fa2 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a3d46aeb-883d-46d9-ac14-4df4aa4cd093_isrForCrosstalkSources_855235_52.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a3d46aeb-883d-46d9-ac14-4df4aa4cd093_isrForCrosstalkSources_855235_52.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash f5d6fb2c823896648f313897d936586e parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/42ff4bce-fd5d-4536-93cb-0c6ad613bdc4_isrForCrosstalkSources_855236_18.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/42ff4bce-fd5d-4536-93cb-0c6ad613bdc4_isrForCrosstalkSources_855236_18.stderr parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash e9fe5acc94ca0e34f2fbe6013721bd96 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7d3d3c09-0649-40e2-95a3-550848722858_isrForCrosstalkSources_855232_18.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7d3d3c09-0649-40e2-95a3-550848722858_isrForCrosstalkSources_855232_18.stderr parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash e6128fe77a4474b0d796a59f00489e7c parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4f4d5892-bf95-4154-9a86-94dafaea5cd4_isrForCrosstalkSources_855240_8.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4f4d5892-bf95-4154-9a86-94dafaea5cd4_isrForCrosstalkSources_855240_8.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash e465368afb2726c4b4f2e86c51d89fbf parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/744ff8da-11b0-45f0-b4eb-6d37956837e7_isrForCrosstalkSources_855232_49.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/744ff8da-11b0-45f0-b4eb-6d37956837e7_isrForCrosstalkSources_855232_49.stderr parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 2d22c451efa3cc1c3988fb81ce59993e parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6295aa04-26f5-4e31-b274-147022823de4_isrForCrosstalkSources_855235_22.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6295aa04-26f5-4e31-b274-147022823de4_isrForCrosstalkSources_855235_22.stderr parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 21b86b4a4c0bf793e86697da97761a96 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ef5b24ec-abb8-4761-a8bb-f4123b62a446_isrForCrosstalkSources_855242_51.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ef5b24ec-abb8-4761-a8bb-f4123b62a446_isrForCrosstalkSources_855242_51.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash f0011f09f0754aa7e86f302917ada76e parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/9bbed066-1fab-4862-8965-cad677d0a454_isrForCrosstalkSources_855155_33.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/9bbed066-1fab-4862-8965-cad677d0a454_isrForCrosstalkSources_855155_33.stderr parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash ba4137576f3c30ab83434b1925de959d parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914621 proc_lsst.multi INFO: [multi] job_id 16914621 2024-03-08 10:00:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 10:00:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914621 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914621 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16914621 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-08 10:00:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-08 10:00:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 10:00:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/39497209-0e56-4755-b056-48164221b4d6_isrForCrosstalkSources_855154_49.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/39497209-0e56-4755-b056-48164221b4d6_isrForCrosstalkSources_855154_49.stderr parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash b9b186627934244e22b0b5c4e3ee77d2 parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4a1c5398-ce76-488f-953d-dfa6a7dbdb00_isrForCrosstalkSources_855240_44.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4a1c5398-ce76-488f-953d-dfa6a7dbdb00_isrForCrosstalkSources_855240_44.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash ec21ebbe5b7f3cc41b385a8cc609b396 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d8f6099a-69f4-46b2-97cc-ce746279a1c9_isrForCrosstalkSources_855237_43.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d8f6099a-69f4-46b2-97cc-ce746279a1c9_isrForCrosstalkSources_855237_43.stderr parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash da893464d4ba393bc924ee63bfc20dfd parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/797b4129-cdb2-4369-9590-44b46942242b_isrForCrosstalkSources_855234_31.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/797b4129-cdb2-4369-9590-44b46942242b_isrForCrosstalkSources_855234_31.stderr parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 3d4addff8cc43b6d9c9f017e7a8907de parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/04de2aac-bc8e-4647-8c8d-b09aa559155a_isrForCrosstalkSources_855238_9.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/04de2aac-bc8e-4647-8c8d-b09aa559155a_isrForCrosstalkSources_855238_9.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 5bd8b1f2bf3c87e24bdc53c817846bdc parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9d2d872-807b-4cb1-86e9-8b55ea7f11df_isrForCrosstalkSources_855231_3.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9d2d872-807b-4cb1-86e9-8b55ea7f11df_isrForCrosstalkSources_855231_3.stderr parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 109524a2cbb97a9579c01a87efe1301e parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d1201382-ef02-4b87-b365-4dd5a3755329_isrForCrosstalkSources_855235_53.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d1201382-ef02-4b87-b365-4dd5a3755329_isrForCrosstalkSources_855235_53.stderr parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash a410a828f5ba88c22b9ae1f55f2cd28d parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b45a4dc0-5d59-4ec6-8611-78ad8d145138_isrForCrosstalkSources_855233_30.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b45a4dc0-5d59-4ec6-8611-78ad8d145138_isrForCrosstalkSources_855233_30.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash b9ddb4132ab76b50a237bfef08d5526f parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c5d6fb80-765c-4f3d-8b0f-799bd1fdcd80_isrForCrosstalkSources_855232_56.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c5d6fb80-765c-4f3d-8b0f-799bd1fdcd80_isrForCrosstalkSources_855232_56.stderr parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 95df4301655c479637a163c2724098f1 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8f8215ad-bb0a-4635-85f7-3691446b3dcf_isrForCrosstalkSources_855233_8.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8f8215ad-bb0a-4635-85f7-3691446b3dcf_isrForCrosstalkSources_855233_8.stderr parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash c98691e953a58c06ee5f6b98150fb755 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/fa651f48-51bb-40e3-bdd6-e9be5f861044_isrForCrosstalkSources_855242_34.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/fa651f48-51bb-40e3-bdd6-e9be5f861044_isrForCrosstalkSources_855242_34.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 280c71f248b07709dc69bbd92a2a1690 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c2afd191-016f-4fe9-8ffd-c7aceca1b024_isrForCrosstalkSources_855242_40.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c2afd191-016f-4fe9-8ffd-c7aceca1b024_isrForCrosstalkSources_855242_40.stderr parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 1ac18733b43678d00012575dbbc42511 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6ab13898-08ea-44fd-9b8f-2c3a71ccb79b_isrForCrosstalkSources_855232_50.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6ab13898-08ea-44fd-9b8f-2c3a71ccb79b_isrForCrosstalkSources_855232_50.stderr parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash f799185e7813a95759ee1e1cfe33964b parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a0546e93-5f21-4633-afe9-11045febc5de_isrForCrosstalkSources_855235_20.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a0546e93-5f21-4633-afe9-11045febc5de_isrForCrosstalkSources_855235_20.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash eb2aac6e4fca42a04c70b8f4069d2192 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/1e9de8e0-2769-46c4-bc03-566a4b79c177_isrForCrosstalkSources_855236_21.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/1e9de8e0-2769-46c4-bc03-566a4b79c177_isrForCrosstalkSources_855236_21.stderr parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash c8243f67a7682788ae1aaf0757696f05 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/869b5115-352e-4617-8575-ef7f092f5ec7_isrForCrosstalkSources_855154_36.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/869b5115-352e-4617-8575-ef7f092f5ec7_isrForCrosstalkSources_855154_36.stderr parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 1e94a1fc63c661c66e63932433c13d42 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c97a9dfd-bddb-4915-9f2b-79eee77a3ecb_isrForCrosstalkSources_855240_35.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c97a9dfd-bddb-4915-9f2b-79eee77a3ecb_isrForCrosstalkSources_855240_35.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 20895f19b7ce86da6315338b73e31a0e parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ec196fdf-72ef-4a8e-87c2-e299064542fe_isrForCrosstalkSources_855235_59.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ec196fdf-72ef-4a8e-87c2-e299064542fe_isrForCrosstalkSources_855235_59.stderr parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 805bad87124abcd96fb4fb64ec2fef67 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-08 10:00:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914622 proc_lsst.multi INFO: [multi] job_id 16914622 2024-03-08 10:00:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 10:00:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914622 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914622 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16914622 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-08 10:00:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-08 10:00:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 10:00:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/424d13d9-6960-4498-91f1-ac709db922d9_isrForCrosstalkSources_855241_16.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/424d13d9-6960-4498-91f1-ac709db922d9_isrForCrosstalkSources_855241_16.stderr parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 77b757a878714b76b4fdb3712443f53d parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ccf298f5-eb27-4d7a-ab82-a9a4c108c3a7_isrForCrosstalkSources_855231_9.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ccf298f5-eb27-4d7a-ab82-a9a4c108c3a7_isrForCrosstalkSources_855231_9.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 97a6f8b16213a30674a1229d937154c6 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d4a4efb6-26cf-4b4b-9e10-d91f208aed21_isrForCrosstalkSources_855156_40.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d4a4efb6-26cf-4b4b-9e10-d91f208aed21_isrForCrosstalkSources_855156_40.stderr parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 445b56ef8b7044b993615380be5eda9d parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e8e3cdb4-cde4-49f7-8478-883d06e90fbe_isrForCrosstalkSources_855154_43.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e8e3cdb4-cde4-49f7-8478-883d06e90fbe_isrForCrosstalkSources_855154_43.stderr parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash bd4e941ff16afd54e2e8c2ce7cf78c9e parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/78cb3f40-51f9-4705-bcf5-005e4318cf6b_isrForCrosstalkSources_855240_17.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/78cb3f40-51f9-4705-bcf5-005e4318cf6b_isrForCrosstalkSources_855240_17.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 43ab4ca98c7cf5045b5a61e9dfc9423d parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/461a93f5-30b4-4b8a-b6d6-10166bcae352_isrForCrosstalkSources_855156_11.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/461a93f5-30b4-4b8a-b6d6-10166bcae352_isrForCrosstalkSources_855156_11.stderr parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash d851511d08e3c713316cf8ed966fa7fb parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9ae9f7e-161b-43d2-99b5-2d8c8ead88fa_isrForCrosstalkSources_855156_24.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9ae9f7e-161b-43d2-99b5-2d8c8ead88fa_isrForCrosstalkSources_855156_24.stderr parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 046ed12c3ac1837f68dc1301ab0c9aae parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/f58dbc77-4ac3-4d3b-8d81-4833c77a1da2_isrForCrosstalkSources_855237_42.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/f58dbc77-4ac3-4d3b-8d81-4833c77a1da2_isrForCrosstalkSources_855237_42.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 99b35aed0abb0efa0b9d37109c30cedf parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/1b28ef59-77c3-4f0a-bad5-0450b4515f63_isrForCrosstalkSources_855242_14.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/1b28ef59-77c3-4f0a-bad5-0450b4515f63_isrForCrosstalkSources_855242_14.stderr parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 7ffd628e50504aee0a53a0ad90279b8d parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/3b1a10cf-a548-4080-8349-e37ba090e3ad_isrForCrosstalkSources_855242_27.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/3b1a10cf-a548-4080-8349-e37ba090e3ad_isrForCrosstalkSources_855242_27.stderr parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 012ce66cf6a80de6d6a408d516858222 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b75c0d10-6280-4979-b14a-5fbc4e14ebcb_isrForCrosstalkSources_855241_14.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b75c0d10-6280-4979-b14a-5fbc4e14ebcb_isrForCrosstalkSources_855241_14.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash a0128389b433c70cf7d183110125744c parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/0ed09341-f3b3-4864-893f-4bba908574a1_isrForCrosstalkSources_855233_47.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/0ed09341-f3b3-4864-893f-4bba908574a1_isrForCrosstalkSources_855233_47.stderr parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 723250f755597a7147bf22aa99aeff07 parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9ffd4e91-19ad-48d8-b280-29eadc69e0e1_isrForCrosstalkSources_855239_3.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9ffd4e91-19ad-48d8-b280-29eadc69e0e1_isrForCrosstalkSources_855239_3.stderr parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash f0c389cb0ebaa044f55c4f629aa5f6f4 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/000ae0b2-f965-433a-9b25-808b14a70e04_isrForCrosstalkSources_855242_37.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/000ae0b2-f965-433a-9b25-808b14a70e04_isrForCrosstalkSources_855242_37.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 631ee183dfc996b8c58f2d7fbff25d94 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/884e2e18-ae29-4955-bfc2-73ee5ae4ad0b_isrForCrosstalkSources_855241_15.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/884e2e18-ae29-4955-bfc2-73ee5ae4ad0b_isrForCrosstalkSources_855241_15.stderr parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 5a6a99dad166159e8814b64ecdf2db50 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d72747a4-6049-4231-b323-1215c521b619_isrForCrosstalkSources_855233_43.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d72747a4-6049-4231-b323-1215c521b619_isrForCrosstalkSources_855233_43.stderr parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 1f2a1191deb1fbcd043d566d803fd16b parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1f3757d0-b725-4b48-88ae-36be11d82590_isrForCrosstalkSources_855240_49.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1f3757d0-b725-4b48-88ae-36be11d82590_isrForCrosstalkSources_855240_49.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 06329364db669de129a0997cd36254ee parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/23df32e9-8371-4e6c-9ac9-39bb09904c73_isrForCrosstalkSources_855232_37.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/23df32e9-8371-4e6c-9ac9-39bb09904c73_isrForCrosstalkSources_855232_37.stderr parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash f467a058f33eb16a1a8d5cf664f47ed7 parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bada7eae-9b21-43e3-b403-8479f55da098_isrForCrosstalkSources_855237_58.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bada7eae-9b21-43e3-b403-8479f55da098_isrForCrosstalkSources_855237_58.stderr parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 1f2b923167176602900896b4f55786e3 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/784f6bc2-f984-426a-8d2a-011caf81b476_isrForCrosstalkSources_855239_24.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/784f6bc2-f984-426a-8d2a-011caf81b476_isrForCrosstalkSources_855239_24.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 58aca6dd8162ec376ea53a4477279640 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/32df5d95-0335-4ff6-b235-5f4cac34d027_isrForCrosstalkSources_855156_27.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/32df5d95-0335-4ff6-b235-5f4cac34d027_isrForCrosstalkSources_855156_27.stderr parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 0e03db13b2402c6151564539f0a4bb75 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2ad0345-0675-4c1f-9891-78ecacc036c4_isrForCrosstalkSources_855239_40.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2ad0345-0675-4c1f-9891-78ecacc036c4_isrForCrosstalkSources_855239_40.stderr parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 53fb71d992d9660328dc2b62a5eea3ba parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/2b75123a-3afb-4377-b4a1-c04fa3a8ac23_isrForCrosstalkSources_855237_1.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/2b75123a-3afb-4377-b4a1-c04fa3a8ac23_isrForCrosstalkSources_855237_1.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 9de8d85dbca971ad1ea074aa9472ac8e parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/639be7b4-6dc7-4df8-86eb-039dbe39d80c_isrForCrosstalkSources_855239_45.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/639be7b4-6dc7-4df8-86eb-039dbe39d80c_isrForCrosstalkSources_855239_45.stderr parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 88f69e8837e1b924f3c20e91ef5bd2fe parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5d1a0095-614c-4458-83fe-b9ead8fe0d13_isrForCrosstalkSources_855156_15.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5d1a0095-614c-4458-83fe-b9ead8fe0d13_isrForCrosstalkSources_855156_15.stderr parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 5bd1dc84e7b717e0cc7857f0e68ca194 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 922 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914623 proc_lsst.multi INFO: [multi] job_id 16914623 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914623 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914623 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/04e3518b-dbba-4e18-81ff-4933195b672b_isrForCrosstalkSources_855156_22.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/04e3518b-dbba-4e18-81ff-4933195b672b_isrForCrosstalkSources_855156_22.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash b77bf17fc46c7f88e083869289d65bc0 parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528ed4e5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3c03f5d1-2e27-42ca-b433-af7bda4b998d_isrForCrosstalkSources_855241_22.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3c03f5d1-2e27-42ca-b433-af7bda4b998d_isrForCrosstalkSources_855241_22.stderr parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16914623 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash ee921641ac6eb1a8d51c56e8284052c4 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfdbb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/90e53691-4aa3-4e03-969d-adef868847b2_isrForCrosstalkSources_855242_2.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/90e53691-4aa3-4e03-969d-adef868847b2_isrForCrosstalkSources_855242_2.stderr parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash ad43df7aa35c16005c19ccbae0ff9217 parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c7b0f8b2-03e8-4449-898f-182108b30d75_isrForCrosstalkSources_855237_16.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c7b0f8b2-03e8-4449-898f-182108b30d75_isrForCrosstalkSources_855237_16.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash e514098b387057791b6ef2e5042d6827 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d4db7c2b-1586-401e-82c3-ec3310cc66c9_isrForCrosstalkSources_855154_22.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d4db7c2b-1586-401e-82c3-ec3310cc66c9_isrForCrosstalkSources_855154_22.stderr parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash f122d202c2e3182166a7ed31d2e68eea parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3c242d9-9990-4680-9ee5-14a2dc487bb4_isrForCrosstalkSources_855235_57.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3c242d9-9990-4680-9ee5-14a2dc487bb4_isrForCrosstalkSources_855235_57.stderr parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash acbc3827ddfd42fc9895530c1dc6d113 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/34ab5a60-a8e2-4887-9195-d9d83982e3ad_isrForCrosstalkSources_855241_3.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/34ab5a60-a8e2-4887-9195-d9d83982e3ad_isrForCrosstalkSources_855241_3.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash d0f8ccf346c89c8172e2c990a45c6960 parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/781649e4-c586-45da-a3ed-34c76c073105_isrForCrosstalkSources_855239_25.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/781649e4-c586-45da-a3ed-34c76c073105_isrForCrosstalkSources_855239_25.stderr parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash e5a9bc64bef61c8e58204a50f2bb4eb4 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1528dfb0e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2b5c483b-699f-4eeb-82f0-2f6049a35898_isrForCrosstalkSources_855240_55.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2b5c483b-699f-4eeb-82f0-2f6049a35898_isrForCrosstalkSources_855240_55.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914624 proc_lsst.multi INFO: [multi] job_id 16914624 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914624 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914624 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16914624 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914625 proc_lsst.multi INFO: [multi] job_id 16914625 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914625 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914625 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16914625 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914626 proc_lsst.multi INFO: [multi] job_id 16914626 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914626 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914626 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16914626 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914627 proc_lsst.multi INFO: [multi] job_id 16914627 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914627 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914627 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16914627 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914628 proc_lsst.multi INFO: [multi] job_id 16914628 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914628 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914628 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16914628 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914629 proc_lsst.multi INFO: [multi] job_id 16914629 2024-03-08 10:00:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16914629 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16914629 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16914629 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-08 10:00:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 10:00:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914630 proc_lsst.multi INFO: [multi] job_id 16914630 2024-03-08 10:00:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16914630 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16914630 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16914630 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-08 10:00:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914631 proc_lsst.multi INFO: [multi] job_id 16914631 2024-03-08 10:00:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:00:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16914631 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16914631 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16914631 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-08 10:00:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914632 proc_lsst.multi INFO: [multi] job_id 16914632 2024-03-08 10:00:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:00:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16914632 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16914632 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16914632 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-08 10:00:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914633 proc_lsst.multi INFO: [multi] job_id 16914633 2024-03-08 10:00:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:00:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16914633 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16914633 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16914633 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-08 10:00:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914634 proc_lsst.multi INFO: [multi] job_id 16914634 2024-03-08 10:00:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16914634 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16914634 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16914634 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-08 10:00:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 10:00:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:57 proc_lsst.multi:170 [INFO] [multi] job_id 16914635 proc_lsst.multi INFO: [multi] job_id 16914635 2024-03-08 10:00:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914635 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914635 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16914635 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-08 10:00:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:57 proc_lsst.multi:170 [INFO] [multi] job_id 16914636 proc_lsst.multi INFO: [multi] job_id 16914636 2024-03-08 10:00:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:00:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914636 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914636 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16914636 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23265528657488 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 38, active_tasks = 930 parsl.dataflow.strategy DEBUG: Executor multi has 930 active tasks, 1/37 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 38 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 195 more blocks parsl.executors.status_handling INFO: Scaling out by 195 blocks parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-08 10:00:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:57 proc_lsst.multi:170 [INFO] [multi] job_id 16914637 proc_lsst.multi INFO: [multi] job_id 16914637 2024-03-08 10:00:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:00:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914637 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914637 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16914637 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-08 10:00:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:57 proc_lsst.multi:170 [INFO] [multi] job_id 16914638 proc_lsst.multi INFO: [multi] job_id 16914638 2024-03-08 10:00:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:00:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914638 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914638 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16914638 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-08 10:00:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:00:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:58 proc_lsst.multi:170 [INFO] [multi] job_id 16914639 proc_lsst.multi INFO: [multi] job_id 16914639 2024-03-08 10:00:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914639 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914639 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16914639 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-08 10:00:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:58 proc_lsst.multi:170 [INFO] [multi] job_id 16914640 proc_lsst.multi INFO: [multi] job_id 16914640 2024-03-08 10:00:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 10:00:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:58 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914640 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914640 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16914640 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-08 10:00:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 10:00:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:59 proc_lsst.multi:170 [INFO] [multi] job_id 16914641 proc_lsst.multi INFO: [multi] job_id 16914641 2024-03-08 10:00:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 10:00:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914641 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914641 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16914641 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-08 10:00:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:00:59 proc_lsst.multi:170 [INFO] [multi] job_id 16914642 proc_lsst.multi INFO: [multi] job_id 16914642 2024-03-08 10:00:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 10:00:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:59 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914642 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914642 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16914642 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-08 10:00:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:00:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:00:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 10:00:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:00:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af3624e88948c922dd6b79f9e2cbfc07 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7d10a525-03f1-4ba9-af1c-737542e95a97_isrForCrosstalkSources_855237_12.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7d10a525-03f1-4ba9-af1c-737542e95a97_isrForCrosstalkSources_855237_12.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 024ca0efd179061c73574679d023403b with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a4579e50-863e-48ec-a408-387570909876_isrForCrosstalkSources_855236_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a4579e50-863e-48ec-a408-387570909876_isrForCrosstalkSources_855236_47.stderr 2024-03-08 10:01:01 proc_lsst.multi:170 [INFO] [multi] job_id 16914643 proc_lsst.multi INFO: [multi] job_id 16914643 2024-03-08 10:01:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 10:01:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:01 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914643 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914643 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16914643 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-08 10:01:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-08 10:01:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 10:01:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:02 proc_lsst.multi:170 [INFO] [multi] job_id 16914647 proc_lsst.multi INFO: [multi] job_id 16914647 2024-03-08 10:01:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 10:01:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914647 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914647 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16914647 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-08 10:01:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:02 proc_lsst.multi:170 [INFO] [multi] job_id 16914648 proc_lsst.multi INFO: [multi] job_id 16914648 2024-03-08 10:01:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 10:01:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:02 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914648 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914648 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16914648 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-08 10:01:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 10:01:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:03 proc_lsst.multi:170 [INFO] [multi] job_id 16914649 proc_lsst.multi INFO: [multi] job_id 16914649 2024-03-08 10:01:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:01:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:03 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914649 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914649 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16914649 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-08 10:01:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-08 10:01:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:01:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30a2cf08f035df85367f18b9afe31ad2 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/84b56d94-7d18-4e3e-aa47-2eb3fbf155f1_isrForCrosstalkSources_855237_13.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/84b56d94-7d18-4e3e-aa47-2eb3fbf155f1_isrForCrosstalkSources_855237_13.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 741c1e90b8a3cf7a08e9b92d291537de with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7d45c4a6-93c6-4218-80d3-8ae959ae6763_isrForCrosstalkSources_855234_26.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7d45c4a6-93c6-4218-80d3-8ae959ae6763_isrForCrosstalkSources_855234_26.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3b2b59a6fbd454fc7ff4f186fa15f83 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0dd4a5d4-9302-4a37-af61-5293233f4e90_isrForCrosstalkSources_855156_34.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0dd4a5d4-9302-4a37-af61-5293233f4e90_isrForCrosstalkSources_855156_34.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5f9832a3628be7527040bd718c08ea with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6f9baa4b-e779-4cc1-8cbc-f964a0adb228_isrForCrosstalkSources_855241_20.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6f9baa4b-e779-4cc1-8cbc-f964a0adb228_isrForCrosstalkSources_855241_20.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23194809df74c85b2b3f49a4e4c27757 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/227654c3-21e6-4857-b815-99369c113585_isrForCrosstalkSources_855233_38.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/227654c3-21e6-4857-b815-99369c113585_isrForCrosstalkSources_855233_38.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8120c5a05ad551dcc1f188cffbb34d0c with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c6a31c6c-77d9-4bef-a302-35e2057688a0_isrForCrosstalkSources_855242_38.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c6a31c6c-77d9-4bef-a302-35e2057688a0_isrForCrosstalkSources_855242_38.stderr parsl.channels.local.local WARNING: Execution of command 'sbatch /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/submit_scripts/parsl.multi.block-48.1709920863.4645212.submit' failed due to Command 'sbatch /mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/submit_scripts/parsl.multi.block-48.1709920863.4645212.submit' timed out after 10 seconds parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-08 10:01:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-08 10:01:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:01:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:14 proc_lsst.multi:170 [INFO] [multi] job_id 16914650 proc_lsst.multi INFO: [multi] job_id 16914650 2024-03-08 10:01:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 10:01:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914650 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914650 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16914650 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-08 10:01:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:14 proc_lsst.multi:170 [INFO] [multi] job_id 16914651 proc_lsst.multi INFO: [multi] job_id 16914651 2024-03-08 10:01:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 10:01:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914651 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914651 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16914651 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-08 10:01:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 10:01:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:15 proc_lsst.multi:170 [INFO] [multi] job_id 16914652 proc_lsst.multi INFO: [multi] job_id 16914652 2024-03-08 10:01:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 10:01:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914652 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914652 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16914652 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-08 10:01:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:15 proc_lsst.multi:170 [INFO] [multi] job_id 16914654 proc_lsst.multi INFO: [multi] job_id 16914654 2024-03-08 10:01:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 10:01:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914654 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914654 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16914654 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-08 10:01:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 10:01:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:16 proc_lsst.multi:170 [INFO] [multi] job_id 16914655 proc_lsst.multi INFO: [multi] job_id 16914655 2024-03-08 10:01:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 10:01:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914655 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914655 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16914655 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-08 10:01:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:16 proc_lsst.multi:170 [INFO] [multi] job_id 16914656 proc_lsst.multi INFO: [multi] job_id 16914656 2024-03-08 10:01:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 10:01:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914656 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914656 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16914656 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-08 10:01:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80143c8ab79b51f178aef288add1d722 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/31e46ff2-767c-49e2-a793-afdbc6904947_isrForCrosstalkSources_855242_11.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/31e46ff2-767c-49e2-a793-afdbc6904947_isrForCrosstalkSources_855242_11.stderr 2024-03-08 10:01:16 proc_lsst.multi:170 [INFO] [multi] job_id 16914657 proc_lsst.multi INFO: [multi] job_id 16914657 2024-03-08 10:01:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 10:01:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914657 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914657 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16914657 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-08 10:01:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 10:01:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:17 proc_lsst.multi:170 [INFO] [multi] job_id 16914658 proc_lsst.multi INFO: [multi] job_id 16914658 2024-03-08 10:01:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16914658 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16914658 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16914658 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-08 10:01:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:17 proc_lsst.multi:170 [INFO] [multi] job_id 16914659 proc_lsst.multi INFO: [multi] job_id 16914659 2024-03-08 10:01:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914659 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914659 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16914659 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-08 10:01:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea09934e841537b655528e7d3fdae735 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/78ed1b05-8cf5-48e8-bafa-316e79570910_isrForCrosstalkSources_855241_25.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/78ed1b05-8cf5-48e8-bafa-316e79570910_isrForCrosstalkSources_855241_25.stderr 2024-03-08 10:01:18 proc_lsst.multi:170 [INFO] [multi] job_id 16914660 proc_lsst.multi INFO: [multi] job_id 16914660 2024-03-08 10:01:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:01:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914660 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914660 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16914660 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-08 10:01:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:18 proc_lsst.multi:170 [INFO] [multi] job_id 16914661 proc_lsst.multi INFO: [multi] job_id 16914661 2024-03-08 10:01:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:01:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914661 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914661 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16914661 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-08 10:01:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-08 10:01:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:19 proc_lsst.multi:170 [INFO] [multi] job_id 16914662 proc_lsst.multi INFO: [multi] job_id 16914662 2024-03-08 10:01:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:01:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914662 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914662 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16914662 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-08 10:01:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:19 proc_lsst.multi:170 [INFO] [multi] job_id 16914663 proc_lsst.multi INFO: [multi] job_id 16914663 2024-03-08 10:01:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914663 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914663 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16914663 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-08 10:01:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:20 proc_lsst.multi:170 [INFO] [multi] job_id 16914664 proc_lsst.multi INFO: [multi] job_id 16914664 2024-03-08 10:01:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 10:01:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914664 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914664 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16914664 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-08 10:01:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 576266cd0ffe66331c89747710519def with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5de965d7-68bd-4e3c-9581-04fafc935684_isrForCrosstalkSources_855233_16.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5de965d7-68bd-4e3c-9581-04fafc935684_isrForCrosstalkSources_855233_16.stderr 2024-03-08 10:01:20 proc_lsst.multi:170 [INFO] [multi] job_id 16914665 proc_lsst.multi INFO: [multi] job_id 16914665 2024-03-08 10:01:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 10:01:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914665 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914665 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16914665 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-08 10:01:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-08 10:01:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66fe6e27e57920ee6435ae81e75dc8a2 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/fd49db3f-a198-4041-b977-5ca699e9962e_isrForCrosstalkSources_855236_17.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/fd49db3f-a198-4041-b977-5ca699e9962e_isrForCrosstalkSources_855236_17.stderr 2024-03-08 10:01:21 proc_lsst.multi:170 [INFO] [multi] job_id 16914666 proc_lsst.multi INFO: [multi] job_id 16914666 2024-03-08 10:01:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 10:01:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914666 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914666 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16914666 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-08 10:01:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:21 proc_lsst.multi:170 [INFO] [multi] job_id 16914667 proc_lsst.multi INFO: [multi] job_id 16914667 2024-03-08 10:01:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 10:01:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914667 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914667 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16914667 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-08 10:01:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-08 10:01:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:22 proc_lsst.multi:170 [INFO] [multi] job_id 16914668 proc_lsst.multi INFO: [multi] job_id 16914668 2024-03-08 10:01:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 10:01:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914668 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914668 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16914668 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-08 10:01:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:22 proc_lsst.multi:170 [INFO] [multi] job_id 16914669 proc_lsst.multi INFO: [multi] job_id 16914669 2024-03-08 10:01:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 10:01:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914669 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914669 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16914669 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-08 10:01:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:22 proc_lsst.multi:170 [INFO] [multi] job_id 16914670 proc_lsst.multi INFO: [multi] job_id 16914670 2024-03-08 10:01:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:01:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914670 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914670 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16914670 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-08 10:01:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-08 10:01:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:23 proc_lsst.multi:170 [INFO] [multi] job_id 16914671 proc_lsst.multi INFO: [multi] job_id 16914671 2024-03-08 10:01:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 10:01:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914671 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914671 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16914671 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-08 10:01:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:23 proc_lsst.multi:170 [INFO] [multi] job_id 16914672 proc_lsst.multi INFO: [multi] job_id 16914672 2024-03-08 10:01:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 10:01:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914672 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914672 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16914672 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-08 10:01:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-08 10:01:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b55111042f9c3acb2f92bcd534d88da5 with result from task 49 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c5f85583-04b3-4d46-b357-b3da66ee461b_isrForCrosstalkSources_855155_36.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c5f85583-04b3-4d46-b357-b3da66ee461b_isrForCrosstalkSources_855155_36.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca9da9115232d844e24b3782bfba6203 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/429c1b10-c35b-4483-b36f-baa7af2b7c55_isrForCrosstalkSources_855241_29.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/429c1b10-c35b-4483-b36f-baa7af2b7c55_isrForCrosstalkSources_855241_29.stderr 2024-03-08 10:01:24 proc_lsst.multi:170 [INFO] [multi] job_id 16914673 proc_lsst.multi INFO: [multi] job_id 16914673 2024-03-08 10:01:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 10:01:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914673 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914673 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16914673 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-08 10:01:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:24 proc_lsst.multi:170 [INFO] [multi] job_id 16914674 proc_lsst.multi INFO: [multi] job_id 16914674 2024-03-08 10:01:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 10:01:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914674 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914674 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16914674 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-08 10:01:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-08 10:01:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:25 proc_lsst.multi:170 [INFO] [multi] job_id 16914675 proc_lsst.multi INFO: [multi] job_id 16914675 2024-03-08 10:01:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 10:01:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914675 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914675 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16914675 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-08 10:01:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:25 proc_lsst.multi:170 [INFO] [multi] job_id 16914677 proc_lsst.multi INFO: [multi] job_id 16914677 2024-03-08 10:01:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 10:01:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914677 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914677 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16914677 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-08 10:01:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-08 10:01:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1606c451c95e3a0c60891867e5e86406 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/482db7ab-ca8c-4190-86b9-f8b74fcbdbed_isrForCrosstalkSources_855237_32.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/482db7ab-ca8c-4190-86b9-f8b74fcbdbed_isrForCrosstalkSources_855237_32.stderr 2024-03-08 10:01:26 proc_lsst.multi:170 [INFO] [multi] job_id 16914678 proc_lsst.multi INFO: [multi] job_id 16914678 2024-03-08 10:01:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 10:01:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914678 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914678 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16914678 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-08 10:01:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-08 10:01:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-08 10:01:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:27 proc_lsst.multi:170 [INFO] [multi] job_id 16914679 proc_lsst.multi INFO: [multi] job_id 16914679 2024-03-08 10:01:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914679 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914679 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16914679 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-08 10:01:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:27 proc_lsst.multi:170 [INFO] [multi] job_id 16914680 proc_lsst.multi INFO: [multi] job_id 16914680 2024-03-08 10:01:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 10:01:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914680 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914680 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16914680 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-08 10:01:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-08 10:01:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9e2b02592bfa9f518e299d60faa41d7 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a4e2fdf7-6838-4401-b138-ba0f80ddb053_isrForCrosstalkSources_855234_60.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a4e2fdf7-6838-4401-b138-ba0f80ddb053_isrForCrosstalkSources_855234_60.stderr 2024-03-08 10:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 16914681 proc_lsst.multi INFO: [multi] job_id 16914681 2024-03-08 10:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 10:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914681 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914681 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 16914681 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-08 10:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 16914682 proc_lsst.multi INFO: [multi] job_id 16914682 2024-03-08 10:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 10:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914682 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914682 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 16914682 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-08 10:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:28 proc_lsst.multi:170 [INFO] [multi] job_id 16914683 proc_lsst.multi INFO: [multi] job_id 16914683 2024-03-08 10:01:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 10:01:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914683 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914683 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 16914683 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-08 10:01:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-08 10:01:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:29 proc_lsst.multi:170 [INFO] [multi] job_id 16914684 proc_lsst.multi INFO: [multi] job_id 16914684 2024-03-08 10:01:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 10:01:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914684 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914684 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 16914684 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-08 10:01:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdd78c672e56379de70199e704bc44ff with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de3251e2-67e5-48da-b4d4-5ed34fc9497c_isrForCrosstalkSources_855233_14.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de3251e2-67e5-48da-b4d4-5ed34fc9497c_isrForCrosstalkSources_855233_14.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eb10d1fe14e29c8eabfb50f1f1f0fc4 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ae4c54b-8880-45a4-b04e-6ce98e25fb5f_isrForCrosstalkSources_855234_41.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ae4c54b-8880-45a4-b04e-6ce98e25fb5f_isrForCrosstalkSources_855234_41.stderr 2024-03-08 10:01:29 proc_lsst.multi:170 [INFO] [multi] job_id 16914685 proc_lsst.multi INFO: [multi] job_id 16914685 2024-03-08 10:01:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 10:01:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914685 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914685 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 16914685 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-08 10:01:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-08 10:01:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:30 proc_lsst.multi:170 [INFO] [multi] job_id 16914686 proc_lsst.multi INFO: [multi] job_id 16914686 2024-03-08 10:01:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 10:01:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914686 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914686 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 16914686 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-08 10:01:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-08 10:01:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-08 10:01:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:31 proc_lsst.multi:170 [INFO] [multi] job_id 16914687 proc_lsst.multi INFO: [multi] job_id 16914687 2024-03-08 10:01:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 10:01:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914687 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914687 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 16914687 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-08 10:01:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:31 proc_lsst.multi:170 [INFO] [multi] job_id 16914688 proc_lsst.multi INFO: [multi] job_id 16914688 2024-03-08 10:01:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 10:01:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914688 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914688 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 16914688 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-08 10:01:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:31 proc_lsst.multi:170 [INFO] [multi] job_id 16914689 proc_lsst.multi INFO: [multi] job_id 16914689 2024-03-08 10:01:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914689 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914689 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 16914689 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-08 10:01:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:32 proc_lsst.multi:170 [INFO] [multi] job_id 16914690 proc_lsst.multi INFO: [multi] job_id 16914690 2024-03-08 10:01:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-08 10:01:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914690 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914690 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 16914690 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-08 10:01:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-08 10:01:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-08 10:01:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48e8eeae26319efe9f091b73898539ec with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/87d2a957-4aff-46b6-87b3-29bb39fd52ba_isrForCrosstalkSources_855242_5.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/87d2a957-4aff-46b6-87b3-29bb39fd52ba_isrForCrosstalkSources_855242_5.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c2342cf4c58d39aa778baac9a641190 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/43d56360-5024-4e89-95d3-3e62830f8439_isrForCrosstalkSources_855238_30.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/43d56360-5024-4e89-95d3-3e62830f8439_isrForCrosstalkSources_855238_30.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d019f6f76d73ebb137c39a4a0c45b7f7 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aafe40dd-d5be-4c95-89bb-3f59de8b3b9b_isrForCrosstalkSources_855233_57.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aafe40dd-d5be-4c95-89bb-3f59de8b3b9b_isrForCrosstalkSources_855233_57.stderr 2024-03-08 10:01:34 proc_lsst.multi:170 [INFO] [multi] job_id 16914691 proc_lsst.multi INFO: [multi] job_id 16914691 2024-03-08 10:01:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-08 10:01:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914691 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914691 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 16914691 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-08 10:01:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-08 10:01:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-08 10:01:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 985279c79a357b06d38c8b739edbe6f9 with result from task 29 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/72f27eb5-1ec9-43e7-8e52-82da0373a0f4_isrForCrosstalkSources_855238_37.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/72f27eb5-1ec9-43e7-8e52-82da0373a0f4_isrForCrosstalkSources_855238_37.stderr 2024-03-08 10:01:35 proc_lsst.multi:170 [INFO] [multi] job_id 16914701 proc_lsst.multi INFO: [multi] job_id 16914701 2024-03-08 10:01:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-08 10:01:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914701 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914701 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 16914701 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-08 10:01:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e7ec2ee5b25cbe1f26c4b8fd4dc642 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8ad96fda-8ed1-4c28-807e-a8463d290eec_isrForCrosstalkSources_855233_54.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8ad96fda-8ed1-4c28-807e-a8463d290eec_isrForCrosstalkSources_855233_54.stderr 2024-03-08 10:01:35 proc_lsst.multi:170 [INFO] [multi] job_id 16914702 proc_lsst.multi INFO: [multi] job_id 16914702 2024-03-08 10:01:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-08 10:01:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914702 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914702 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 16914702 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-08 10:01:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-08 10:01:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:36 proc_lsst.multi:170 [INFO] [multi] job_id 16914703 proc_lsst.multi INFO: [multi] job_id 16914703 2024-03-08 10:01:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-08 10:01:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914703 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914703 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 16914703 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-08 10:01:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:36 proc_lsst.multi:170 [INFO] [multi] job_id 16914704 proc_lsst.multi INFO: [multi] job_id 16914704 2024-03-08 10:01:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-08 10:01:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914704 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914704 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 16914704 parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-08 10:01:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-08 10:01:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9eb2489a5e361b871503cf9db60810 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24149a6f-2557-4cc7-a771-f67fc5463261_isrForCrosstalkSources_855231_59.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24149a6f-2557-4cc7-a771-f67fc5463261_isrForCrosstalkSources_855231_59.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d236cdf831ef06ad2ac8ba142e5eabff with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/cc3155cf-a48b-461e-8934-b4a0b3e3af3c_isrForCrosstalkSources_855236_22.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/cc3155cf-a48b-461e-8934-b4a0b3e3af3c_isrForCrosstalkSources_855236_22.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c33be7ce417c20ad93b5d0bc68b62543 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ab951d3c-e8d5-4c10-9c05-a8683f5bb7b4_isrForCrosstalkSources_855242_20.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ab951d3c-e8d5-4c10-9c05-a8683f5bb7b4_isrForCrosstalkSources_855242_20.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3e5d645c846570a5fb97b2e727b1aa5 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f91041a5-6d40-486e-8d37-eb5b0fc674b0_isrForCrosstalkSources_855241_24.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f91041a5-6d40-486e-8d37-eb5b0fc674b0_isrForCrosstalkSources_855241_24.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0c4cc7a73956fbc93400c8d91bbd8b5 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2e15e4ff-725d-4cb9-9886-a0c4e7a07a8e_isrForCrosstalkSources_855238_7.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2e15e4ff-725d-4cb9-9886-a0c4e7a07a8e_isrForCrosstalkSources_855238_7.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9e68efd9a079eb3fc48c11f2a37ca6 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9421d547-733f-48d7-acf9-eb3154a5a978_isrForCrosstalkSources_855240_56.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9421d547-733f-48d7-acf9-eb3154a5a978_isrForCrosstalkSources_855240_56.stderr 2024-03-08 10:01:37 proc_lsst.multi:170 [INFO] [multi] job_id 16914705 proc_lsst.multi INFO: [multi] job_id 16914705 2024-03-08 10:01:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-08 10:01:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914705 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914705 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 16914705 parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-08 10:01:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ae115a417ba0badb73c67fb74dff4a with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bd11c6bd-d298-4a51-86cb-ae61eba185ae_isrForCrosstalkSources_855240_48.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bd11c6bd-d298-4a51-86cb-ae61eba185ae_isrForCrosstalkSources_855240_48.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dacb519722b2920aa6b9a68b3a960079 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6bad174b-4671-4f47-8823-962fc2a6de56_isrForCrosstalkSources_855241_8.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6bad174b-4671-4f47-8823-962fc2a6de56_isrForCrosstalkSources_855241_8.stderr 2024-03-08 10:01:37 proc_lsst.multi:170 [INFO] [multi] job_id 16914706 proc_lsst.multi INFO: [multi] job_id 16914706 2024-03-08 10:01:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-08 10:01:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914706 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914706 parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 16914706 parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-08 10:01:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-08 10:01:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e513ef24f4e58c910b2fcefb8266a61e with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0fd69046-8072-4ab2-ae05-e22329f672cf_isrForCrosstalkSources_855231_53.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0fd69046-8072-4ab2-ae05-e22329f672cf_isrForCrosstalkSources_855231_53.stderr 2024-03-08 10:01:38 proc_lsst.multi:170 [INFO] [multi] job_id 16914707 proc_lsst.multi INFO: [multi] job_id 16914707 2024-03-08 10:01:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-08 10:01:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914707 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914707 parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 16914707 parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-08 10:01:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:38 proc_lsst.multi:170 [INFO] [multi] job_id 16914708 proc_lsst.multi INFO: [multi] job_id 16914708 2024-03-08 10:01:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-08 10:01:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914708 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914708 parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 16914708 parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-08 10:01:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-08 10:01:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:39 proc_lsst.multi:170 [INFO] [multi] job_id 16914709 proc_lsst.multi INFO: [multi] job_id 16914709 2024-03-08 10:01:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-08 10:01:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914709 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914709 parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 16914709 parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-08 10:01:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9045b159299b659395bdc6c298058065 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/e11a38b3-97be-4386-aa6a-95993bb43c49_isrForCrosstalkSources_855233_37.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/e11a38b3-97be-4386-aa6a-95993bb43c49_isrForCrosstalkSources_855233_37.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4c09b305335157966922d66b2fb9d3 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/67a88562-91d6-44cf-b739-eeea56adba21_isrForCrosstalkSources_855234_52.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/67a88562-91d6-44cf-b739-eeea56adba21_isrForCrosstalkSources_855234_52.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3da8a38bca2250c46989698a4b9ceb65 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/946fd4b9-97b0-429d-8d5b-9e27db30435c_isrForCrosstalkSources_855236_52.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/946fd4b9-97b0-429d-8d5b-9e27db30435c_isrForCrosstalkSources_855236_52.stderr 2024-03-08 10:01:39 proc_lsst.multi:170 [INFO] [multi] job_id 16914710 proc_lsst.multi INFO: [multi] job_id 16914710 2024-03-08 10:01:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-08 10:01:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914710 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914710 parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 16914710 parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-08 10:01:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-08 10:01:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cae465c55922b584c35d2f40a5132a8d with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/ae8ae8b5-f6fd-4165-925b-9dbabbd42515_isrForCrosstalkSources_855232_48.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/ae8ae8b5-f6fd-4165-925b-9dbabbd42515_isrForCrosstalkSources_855232_48.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9baf3a58904d7ad00dc75a22a551b24 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93f965db-cb41-423b-9739-f96ddcc92e6a_isrForCrosstalkSources_855155_52.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93f965db-cb41-423b-9739-f96ddcc92e6a_isrForCrosstalkSources_855155_52.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e7c40e8fe09ac43fca80b89a49532ad with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/47ba14dc-bf3a-4a17-bbd1-db78fb1d2f70_isrForCrosstalkSources_855233_55.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/47ba14dc-bf3a-4a17-bbd1-db78fb1d2f70_isrForCrosstalkSources_855233_55.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67b18df727ea919a608195d96f905270 with result from task 25 2024-03-08 10:01:40 proc_lsst.multi:170 [INFO] [multi] job_id 16914711 proc_lsst.multi INFO: [multi] job_id 16914711 2024-03-08 10:01:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-08 10:01:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914711 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914711 parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 16914711 parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-08 10:01:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/133c84e7-8363-444b-9bed-8eb9d97d25de_isrForCrosstalkSources_855235_26.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/133c84e7-8363-444b-9bed-8eb9d97d25de_isrForCrosstalkSources_855235_26.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 936b4e353e1b75cf24365d0eda91f0b2 with result from task 23 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/980ebe94-d071-42a7-a6c6-06b59b8752db_isrForCrosstalkSources_855240_28.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/980ebe94-d071-42a7-a6c6-06b59b8752db_isrForCrosstalkSources_855240_28.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b5dec039f47ea919a9906969f6b3ccc with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/76bd26a1-1893-4ee0-82a6-bddbd19c80ee_isrForCrosstalkSources_855232_32.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/76bd26a1-1893-4ee0-82a6-bddbd19c80ee_isrForCrosstalkSources_855232_32.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a2177476854818613bbbe8de941c361 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5e998b9e-4555-4708-a9a8-7ce6c31897d6_isrForCrosstalkSources_855234_33.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5e998b9e-4555-4708-a9a8-7ce6c31897d6_isrForCrosstalkSources_855234_33.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ee273b4525897ab2b96ced976015ec with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cb7e334a-4c0c-40aa-9c33-e79a2612e2ab_isrForCrosstalkSources_855241_47.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cb7e334a-4c0c-40aa-9c33-e79a2612e2ab_isrForCrosstalkSources_855241_47.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a37a58edcf10d6ec99db54456777fa74 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/585a4569-84f3-4a96-b4e9-b7ab0cd1f35f_isrForCrosstalkSources_855237_51.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/585a4569-84f3-4a96-b4e9-b7ab0cd1f35f_isrForCrosstalkSources_855237_51.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a9fd8a966bff9a8f34a963ff8e7a54 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/a7b0ead8-6164-4037-8a69-36978ed2f65d_isrForCrosstalkSources_855241_33.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/a7b0ead8-6164-4037-8a69-36978ed2f65d_isrForCrosstalkSources_855241_33.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c07d32ad6ff70f54d42a0b9520041c9 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/629ff215-4e1f-4a8a-aec2-34aa09e46bc3_isrForCrosstalkSources_855240_43.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/629ff215-4e1f-4a8a-aec2-34aa09e46bc3_isrForCrosstalkSources_855240_43.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a626c4c8d74eab2081d5345540bb06c with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/cf299999-c443-4478-9e1c-1fa1d53bdc2c_isrForCrosstalkSources_855242_56.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/cf299999-c443-4478-9e1c-1fa1d53bdc2c_isrForCrosstalkSources_855242_56.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca11a2b6b25e0a4313f23461d39d607 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5cad50eb-bcc8-42ae-9350-9edec3936a91_isrForCrosstalkSources_855236_49.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5cad50eb-bcc8-42ae-9350-9edec3936a91_isrForCrosstalkSources_855236_49.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2611dd1a40957f5e6333443303f878a7 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f07ef528-72db-4507-868e-7d2f86241f90_isrForCrosstalkSources_855156_55.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f07ef528-72db-4507-868e-7d2f86241f90_isrForCrosstalkSources_855156_55.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbf31e56988e5aa522a99cca5e02e29b with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/508a58ce-94b5-4ec1-a9f8-70ed09ed2a1b_isrForCrosstalkSources_855237_26.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/508a58ce-94b5-4ec1-a9f8-70ed09ed2a1b_isrForCrosstalkSources_855237_26.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c172512dca8b331f102376e833e0cae9 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5120dff6-fa56-4dd3-a923-d8d2a6654166_isrForCrosstalkSources_855156_29.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5120dff6-fa56-4dd3-a923-d8d2a6654166_isrForCrosstalkSources_855156_29.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7e6905540f3f2d059fc08ea300c5060 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9cb8593b-497e-4cc7-816f-6b4f5e2419ab_isrForCrosstalkSources_855233_20.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9cb8593b-497e-4cc7-816f-6b4f5e2419ab_isrForCrosstalkSources_855233_20.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0edafa04d36f33a1c2516ff767fbb998 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8446130-c7fd-4171-afcb-30fa7bf8347f_isrForCrosstalkSources_855155_21.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8446130-c7fd-4171-afcb-30fa7bf8347f_isrForCrosstalkSources_855155_21.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd17528b21100b410e97ebe77bd6168c with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/676fecaf-41a1-4b9f-aebb-40c267f13323_isrForCrosstalkSources_855240_38.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/676fecaf-41a1-4b9f-aebb-40c267f13323_isrForCrosstalkSources_855240_38.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d9b8d1508a47d437955788798bec67f with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9f298d1-10c9-4a53-8e91-458555c22574_isrForCrosstalkSources_855231_4.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9f298d1-10c9-4a53-8e91-458555c22574_isrForCrosstalkSources_855231_4.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b82e56dfc2bf8b7e272bd0f52501c4ed with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/53077817-ff60-4e7b-9ba2-159266d69c46_isrForCrosstalkSources_855238_54.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/53077817-ff60-4e7b-9ba2-159266d69c46_isrForCrosstalkSources_855238_54.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf2d52c6a230d798307489f66d88b62 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/31305552-509e-48fa-888c-1d3674bc283e_isrForCrosstalkSources_855233_62.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/31305552-509e-48fa-888c-1d3674bc283e_isrForCrosstalkSources_855233_62.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e59b35245c1818d5e2c3dedd71751a7 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/848be6ef-c7e9-4658-a8eb-e7638fd87d4f_isrForCrosstalkSources_855232_14.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/848be6ef-c7e9-4658-a8eb-e7638fd87d4f_isrForCrosstalkSources_855232_14.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aee0faed429ea9325ebe4436fee6d3be with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bd55e3e5-99e5-4b99-a3b3-ee906cb7ab3f_isrForCrosstalkSources_855236_20.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bd55e3e5-99e5-4b99-a3b3-ee906cb7ab3f_isrForCrosstalkSources_855236_20.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb881a77c66759b3522c1313cebe3f7e with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/53647791-1a75-4ff2-82f9-1248aeb08086_isrForCrosstalkSources_855234_37.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/53647791-1a75-4ff2-82f9-1248aeb08086_isrForCrosstalkSources_855234_37.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0445ab17d37bba4dac3176abea97714d with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d859edb6-6557-4ebe-a0f4-2f4cf9304a07_isrForCrosstalkSources_855154_38.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d859edb6-6557-4ebe-a0f4-2f4cf9304a07_isrForCrosstalkSources_855154_38.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7d685d9ad290e489887fb37563fb19 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2927f888-c00f-4f0c-9754-f065d5cd984f_isrForCrosstalkSources_855238_55.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2927f888-c00f-4f0c-9754-f065d5cd984f_isrForCrosstalkSources_855238_55.stderr 2024-03-08 10:01:40 proc_lsst.multi:170 [INFO] [multi] job_id 16914712 proc_lsst.multi INFO: [multi] job_id 16914712 2024-03-08 10:01:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-08 10:01:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914712 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914712 parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 16914712 parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-08 10:01:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-08 10:01:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:41 proc_lsst.multi:170 [INFO] [multi] job_id 16914713 proc_lsst.multi INFO: [multi] job_id 16914713 2024-03-08 10:01:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-08 10:01:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914713 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914713 parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 16914713 parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-08 10:01:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-08 10:01:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:41 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:41 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:41 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-08 10:01:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:42 proc_lsst.multi:170 [INFO] [multi] job_id 16914714 proc_lsst.multi INFO: [multi] job_id 16914714 2024-03-08 10:01:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-08 10:01:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914714 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914714 parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 16914714 parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-08 10:01:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:42 proc_lsst.multi:170 [INFO] [multi] job_id 16914715 proc_lsst.multi INFO: [multi] job_id 16914715 2024-03-08 10:01:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-08 10:01:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914715 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914715 parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 16914715 parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-08 10:01:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:42 proc_lsst.multi:170 [INFO] [multi] job_id 16914716 proc_lsst.multi INFO: [multi] job_id 16914716 2024-03-08 10:01:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-08 10:01:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914716 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914716 parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 16914716 parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-08 10:01:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-08 10:01:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebd82d95112a3cee46d80657271fe70a with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c2238409-0a02-4056-83ca-b63717ab45d7_isrForCrosstalkSources_855236_4.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c2238409-0a02-4056-83ca-b63717ab45d7_isrForCrosstalkSources_855236_4.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b05d6a283675618d3ee79a784ba16495 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/097dab31-bd07-480b-b08a-028ee1df885b_isrForCrosstalkSources_855239_5.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/097dab31-bd07-480b-b08a-028ee1df885b_isrForCrosstalkSources_855239_5.stderr 2024-03-08 10:01:43 proc_lsst.multi:170 [INFO] [multi] job_id 16914717 proc_lsst.multi INFO: [multi] job_id 16914717 2024-03-08 10:01:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-08 10:01:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:43 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914717 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914717 parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 16914717 parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-08 10:01:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-08 10:01:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:43 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:43 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:43 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-08 10:01:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7314b5f115ca6b1c8af2c5be5eeb8cf7 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8c77b052-22d3-4621-a84e-7262e1fa5f8f_isrForCrosstalkSources_855233_53.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8c77b052-22d3-4621-a84e-7262e1fa5f8f_isrForCrosstalkSources_855233_53.stderr 2024-03-08 10:01:44 proc_lsst.multi:170 [INFO] [multi] job_id 16914718 proc_lsst.multi INFO: [multi] job_id 16914718 2024-03-08 10:01:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-08 10:01:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914718 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914718 parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 16914718 parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-08 10:01:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-08 10:01:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-08 10:01:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:44 proc_lsst.multi:170 [INFO] [multi] job_id 16914719 proc_lsst.multi INFO: [multi] job_id 16914719 2024-03-08 10:01:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-08 10:01:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:45 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914719 parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c33056be7f228fec8ae1c6f3cb5b0b13 with result from task 80 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914719 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 16914719 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.executors.status_handling INFO: Allocated block ID 108 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3e0754d0-967f-4cd9-92b1-5fa9adeb19e3_isrForCrosstalkSources_855156_33.stdout parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-08 10:01:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3e0754d0-967f-4cd9-92b1-5fa9adeb19e3_isrForCrosstalkSources_855156_33.stderr proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-08 10:01:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:45 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:45 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-08 10:01:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:46 proc_lsst.multi:170 [INFO] [multi] job_id 16914720 proc_lsst.multi INFO: [multi] job_id 16914720 2024-03-08 10:01:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-08 10:01:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914720 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914720 parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 16914720 parsl.executors.status_handling INFO: Allocated block ID 109 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-109 2024-03-08 10:01:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-109 2024-03-08 10:01:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-08 10:01:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=109 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914721 proc_lsst.multi INFO: [multi] job_id 16914721 2024-03-08 10:01:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 53 parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56f925546ede14eaf18c1138128d2248 with result from task 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-08 10:01:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/698a20fe-667e-4c21-bd80-434919751799_isrForCrosstalkSources_855240_27.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/698a20fe-667e-4c21-bd80-434919751799_isrForCrosstalkSources_855240_27.stderr proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914721 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914721 parsl.executors.status_handling DEBUG: Launched block 109 on executor multi with job ID 16914721 parsl.executors.status_handling INFO: Allocated block ID 110 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-110 2024-03-08 10:01:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abe9258094a7fe9338947be2eb984820 with result from task 85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-110 2024-03-08 10:01:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/05b0656d-081d-4661-ac23-8af9ac1d5db8_isrForCrosstalkSources_855234_16.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/05b0656d-081d-4661-ac23-8af9ac1d5db8_isrForCrosstalkSources_855234_16.stderr proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-08 10:01:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=110 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b844f41125a084ec3b8e24f31df69fcb with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e293711e-dcd0-41fa-9276-258efbbe40b4_isrForCrosstalkSources_855235_51.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e293711e-dcd0-41fa-9276-258efbbe40b4_isrForCrosstalkSources_855235_51.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaccf7c1d0b980a26badb18f3edc2636 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5dbfd987-a196-40e4-bdd1-13bc7f696985_isrForCrosstalkSources_855242_47.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5dbfd987-a196-40e4-bdd1-13bc7f696985_isrForCrosstalkSources_855242_47.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35769237b2f2b8999143dfda0576f8e2 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/cad6ab34-38ec-4544-8caa-e6d0b81f3e71_isrForCrosstalkSources_855238_1.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/cad6ab34-38ec-4544-8caa-e6d0b81f3e71_isrForCrosstalkSources_855238_1.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c2306478f8aa92768dc3449f810f6b with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/de7f1d17-7825-467f-b440-4932c5b4bc94_isrForCrosstalkSources_855234_24.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/de7f1d17-7825-467f-b440-4932c5b4bc94_isrForCrosstalkSources_855234_24.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ced37790ea6bc76bc9f7f29b78901ae3 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a6f1a15-62e4-4a1a-b714-32df2bff67d8_isrForCrosstalkSources_855235_30.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a6f1a15-62e4-4a1a-b714-32df2bff67d8_isrForCrosstalkSources_855235_30.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ffefea3857830376dab9adacd4f0806 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/87f4d9ac-cd86-497b-b15e-a9c230424cc8_isrForCrosstalkSources_855240_41.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/87f4d9ac-cd86-497b-b15e-a9c230424cc8_isrForCrosstalkSources_855240_41.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83ff54d6d70d4b2c45fe2964120259a5 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/44ae3daa-8271-4e27-be47-2cee65d46f78_isrForCrosstalkSources_855242_61.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/44ae3daa-8271-4e27-be47-2cee65d46f78_isrForCrosstalkSources_855242_61.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf38d8eb6b27e7d2334783584c4a259 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/29ee462d-96dd-442c-964e-a04035a1179d_isrForCrosstalkSources_855231_43.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/29ee462d-96dd-442c-964e-a04035a1179d_isrForCrosstalkSources_855231_43.stderr 2024-03-08 10:01:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914722 proc_lsst.multi INFO: [multi] job_id 16914722 2024-03-08 10:01:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-08 10:01:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914722 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914722 parsl.executors.status_handling DEBUG: Launched block 110 on executor multi with job ID 16914722 parsl.executors.status_handling INFO: Allocated block ID 111 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-111 2024-03-08 10:01:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-111 2024-03-08 10:01:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 54 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 54 2024-03-08 10:01:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=111 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01972f7f11b286b6e1fd5c0ff48ae99b with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/4050e335-f406-4032-83c3-9bf7220bcf26_isrForCrosstalkSources_855241_13.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/4050e335-f406-4032-83c3-9bf7220bcf26_isrForCrosstalkSources_855241_13.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc8f4e40b0e484da7f786b901c37e8cb with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/1cdece92-f054-430e-bbbc-c18276d6904e_isrForCrosstalkSources_855233_29.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/1cdece92-f054-430e-bbbc-c18276d6904e_isrForCrosstalkSources_855233_29.stderr 2024-03-08 10:01:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914723 proc_lsst.multi INFO: [multi] job_id 16914723 2024-03-08 10:01:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-08 10:01:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914723 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914723 parsl.executors.status_handling DEBUG: Launched block 111 on executor multi with job ID 16914723 parsl.executors.status_handling INFO: Allocated block ID 112 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-112 2024-03-08 10:01:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-112 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 55 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 55 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=112 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914724 proc_lsst.multi INFO: [multi] job_id 16914724 2024-03-08 10:01:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-08 10:01:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914724 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914724 parsl.executors.status_handling DEBUG: Launched block 112 on executor multi with job ID 16914724 parsl.executors.status_handling INFO: Allocated block ID 113 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-113 2024-03-08 10:01:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-113 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 56 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 56 2024-03-08 10:01:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=113 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914725 proc_lsst.multi INFO: [multi] job_id 16914725 2024-03-08 10:01:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-08 10:01:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914725 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914725 parsl.executors.status_handling DEBUG: Launched block 113 on executor multi with job ID 16914725 parsl.executors.status_handling INFO: Allocated block ID 114 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-114 2024-03-08 10:01:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-114 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 57 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 57 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=114 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914726 proc_lsst.multi INFO: [multi] job_id 16914726 2024-03-08 10:01:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-08 10:01:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914726 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914726 parsl.executors.status_handling DEBUG: Launched block 114 on executor multi with job ID 16914726 parsl.executors.status_handling INFO: Allocated block ID 115 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-115 2024-03-08 10:01:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-115 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 58 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 58 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=115 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914727 proc_lsst.multi INFO: [multi] job_id 16914727 2024-03-08 10:01:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-08 10:01:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914727 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914727 parsl.executors.status_handling DEBUG: Launched block 115 on executor multi with job ID 16914727 parsl.executors.status_handling INFO: Allocated block ID 116 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-116 2024-03-08 10:01:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-116 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 59 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 59 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=116 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914728 proc_lsst.multi INFO: [multi] job_id 16914728 2024-03-08 10:01:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-08 10:01:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914728 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914728 parsl.executors.status_handling DEBUG: Launched block 116 on executor multi with job ID 16914728 parsl.executors.status_handling INFO: Allocated block ID 117 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-117 2024-03-08 10:01:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-117 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 60 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 60 2024-03-08 10:01:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=117 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914729 proc_lsst.multi INFO: [multi] job_id 16914729 2024-03-08 10:01:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-08 10:01:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914729 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914729 parsl.executors.status_handling DEBUG: Launched block 117 on executor multi with job ID 16914729 parsl.executors.status_handling INFO: Allocated block ID 118 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-118 2024-03-08 10:01:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-118 2024-03-08 10:01:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d644c69e3aa84f3a7af1f1467dc1b6d with result from task 136 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ac1015ae-26d3-41e5-941f-44f6f109d9de_isrForCrosstalkSources_855233_32.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ac1015ae-26d3-41e5-941f-44f6f109d9de_isrForCrosstalkSources_855233_32.stderr proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 61 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 61 2024-03-08 10:01:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94aa0cd9bb8e789a079290b731b55d8c with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/518b2b82-66c0-4f43-877e-03e03ce94ea4_isrForCrosstalkSources_855231_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/518b2b82-66c0-4f43-877e-03e03ce94ea4_isrForCrosstalkSources_855231_23.stderr proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=118 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7f3b8ed276ec5cd13f644de8cacab39 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f55e0944-391b-408d-8098-a7ab7536c42c_isrForCrosstalkSources_855241_30.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f55e0944-391b-408d-8098-a7ab7536c42c_isrForCrosstalkSources_855241_30.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecb4ff51019ca34dc17bac7dcd39fc13 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/486cd0d5-71ed-4781-a4cb-c80f68463cae_isrForCrosstalkSources_855241_55.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/486cd0d5-71ed-4781-a4cb-c80f68463cae_isrForCrosstalkSources_855241_55.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee9051c4b3b66da31f9674cf5b1b1e1 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7751b8b5-0ffd-4847-adaf-a68de266f6dd_isrForCrosstalkSources_855232_24.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7751b8b5-0ffd-4847-adaf-a68de266f6dd_isrForCrosstalkSources_855232_24.stderr 2024-03-08 10:01:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914730 proc_lsst.multi INFO: [multi] job_id 16914730 2024-03-08 10:01:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-08 10:01:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914730 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914730 parsl.executors.status_handling DEBUG: Launched block 118 on executor multi with job ID 16914730 parsl.executors.status_handling INFO: Allocated block ID 119 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-119 2024-03-08 10:01:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-119 2024-03-08 10:01:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 62 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 62 2024-03-08 10:01:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=119 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea71b1af9d7342a117f8076653629348 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/220002d9-0a69-478c-a7d6-cb6fc3f1b4a3_isrForCrosstalkSources_855232_4.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/220002d9-0a69-478c-a7d6-cb6fc3f1b4a3_isrForCrosstalkSources_855232_4.stderr 2024-03-08 10:01:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914731 proc_lsst.multi INFO: [multi] job_id 16914731 2024-03-08 10:01:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-08 10:01:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914731 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914731 parsl.executors.status_handling DEBUG: Launched block 119 on executor multi with job ID 16914731 parsl.executors.status_handling INFO: Allocated block ID 120 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-120 2024-03-08 10:01:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-120 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f121b6b4da8d3ccec97aea7325e80fc8 with result from task 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f42b2cd1-49c4-42ba-a106-cc3c302c3b06_isrForCrosstalkSources_855235_34.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f42b2cd1-49c4-42ba-a106-cc3c302c3b06_isrForCrosstalkSources_855235_34.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5adee9c465744757d4646239bbffa0e with result from task 120 proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 63 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 63 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=120 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9b20e6d0-9a91-4811-93bf-5dbce9bfd85b_isrForCrosstalkSources_855233_11.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9b20e6d0-9a91-4811-93bf-5dbce9bfd85b_isrForCrosstalkSources_855233_11.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1599178ed4a0b017dd663a696d6e84e0 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/74f67774-abb8-4d1b-9005-828fc172781e_isrForCrosstalkSources_855233_28.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/74f67774-abb8-4d1b-9005-828fc172781e_isrForCrosstalkSources_855233_28.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec705d257445ce67f0d10e2ebe593d92 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ea98ee2-d12c-44f8-98a9-9907814bb944_isrForCrosstalkSources_855236_26.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ea98ee2-d12c-44f8-98a9-9907814bb944_isrForCrosstalkSources_855236_26.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e977e5afe5b5cb85850ab1df30e978b with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/065fdd40-fae7-4f5f-b201-30a040e1e61e_isrForCrosstalkSources_855239_4.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/065fdd40-fae7-4f5f-b201-30a040e1e61e_isrForCrosstalkSources_855239_4.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee1abf3558974f519f5afd18af8416c1 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/55e84964-809a-480c-b850-adbd2234c0d9_isrForCrosstalkSources_855156_37.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/55e84964-809a-480c-b850-adbd2234c0d9_isrForCrosstalkSources_855156_37.stderr 2024-03-08 10:01:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914732 proc_lsst.multi INFO: [multi] job_id 16914732 2024-03-08 10:01:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-08 10:01:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914732 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914732 parsl.executors.status_handling DEBUG: Launched block 120 on executor multi with job ID 16914732 parsl.executors.status_handling INFO: Allocated block ID 121 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-121 2024-03-08 10:01:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-121 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 64 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 64 2024-03-08 10:01:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=121 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d896897f9f0c210073e3af979615046 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/73c4facd-6ef2-4a76-8cc7-8e9f3739b549_isrForCrosstalkSources_855156_5.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/73c4facd-6ef2-4a76-8cc7-8e9f3739b549_isrForCrosstalkSources_855156_5.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e4b3bf5d2c0ecb5fcb3c90a6944b5ba with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d3736718-a427-4053-bd96-8b87e546c63a_isrForCrosstalkSources_855154_24.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d3736718-a427-4053-bd96-8b87e546c63a_isrForCrosstalkSources_855154_24.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 547b281fa5b3162ed33fc7d0f5f8a011 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/dee42f89-5c85-4667-b60d-b75a56dddda6_isrForCrosstalkSources_855231_31.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/dee42f89-5c85-4667-b60d-b75a56dddda6_isrForCrosstalkSources_855231_31.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8ceb7ab0f7edb96bac3296f03bbbe6f with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/80f169a8-cc4e-46aa-90de-5219a06b2422_isrForCrosstalkSources_855241_9.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/80f169a8-cc4e-46aa-90de-5219a06b2422_isrForCrosstalkSources_855241_9.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f6938f1afb39ba401560962cd7d4127 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2e6d6b50-8330-4feb-826a-d5b8c95721b7_isrForCrosstalkSources_855155_45.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2e6d6b50-8330-4feb-826a-d5b8c95721b7_isrForCrosstalkSources_855155_45.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9bdc7f709d701511a6ba2fbc8503136 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/0ffb7868-4142-46db-b05d-c8b8376561d9_isrForCrosstalkSources_855235_29.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/0ffb7868-4142-46db-b05d-c8b8376561d9_isrForCrosstalkSources_855235_29.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a8d7b67b86be22c68f71c36bf36073 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e4796a41-1666-4920-91c8-5c6e2287776d_isrForCrosstalkSources_855235_8.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e4796a41-1666-4920-91c8-5c6e2287776d_isrForCrosstalkSources_855235_8.stderr 2024-03-08 10:01:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914733 proc_lsst.multi INFO: [multi] job_id 16914733 2024-03-08 10:01:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-08 10:01:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914733 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914733 parsl.executors.status_handling DEBUG: Launched block 121 on executor multi with job ID 16914733 parsl.executors.status_handling INFO: Allocated block ID 122 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-122 2024-03-08 10:01:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-122 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 65 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 65 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=122 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 762e3f8db23c732c246c2829a3b9ceea with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8fb053b7-aabc-4e3e-9f12-79ae8f3f5091_isrForCrosstalkSources_855232_3.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8fb053b7-aabc-4e3e-9f12-79ae8f3f5091_isrForCrosstalkSources_855232_3.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c574ea27aed944222518a8b424d89e5b with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/64e55775-f9fc-4c5b-85b9-76917a0c5619_isrForCrosstalkSources_855241_52.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/64e55775-f9fc-4c5b-85b9-76917a0c5619_isrForCrosstalkSources_855241_52.stderr 2024-03-08 10:01:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914734 proc_lsst.multi INFO: [multi] job_id 16914734 2024-03-08 10:01:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-08 10:01:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914734 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914734 parsl.executors.status_handling DEBUG: Launched block 122 on executor multi with job ID 16914734 parsl.executors.status_handling INFO: Allocated block ID 123 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-123 2024-03-08 10:01:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-123 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 66 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 66 2024-03-08 10:01:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=123 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ba8bc79d658f07fd036b0bca3c571d5 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e2de5018-3f4d-43d5-b56c-f9b764dd0a45_isrForCrosstalkSources_855238_16.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e2de5018-3f4d-43d5-b56c-f9b764dd0a45_isrForCrosstalkSources_855238_16.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb7c60fe263759b59b2442dda4405135 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d9de2441-c958-4462-b50b-7ccb2f07cdd9_isrForCrosstalkSources_855240_9.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d9de2441-c958-4462-b50b-7ccb2f07cdd9_isrForCrosstalkSources_855240_9.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44d86a05d27eb4e187329fdaabedc32e with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c04ad081-122c-4c14-ba8a-bd4816052fd5_isrForCrosstalkSources_855237_40.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c04ad081-122c-4c14-ba8a-bd4816052fd5_isrForCrosstalkSources_855237_40.stderr 2024-03-08 10:01:57 proc_lsst.multi:170 [INFO] [multi] job_id 16914735 proc_lsst.multi INFO: [multi] job_id 16914735 2024-03-08 10:01:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-08 10:01:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914735 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914735 parsl.executors.status_handling DEBUG: Launched block 123 on executor multi with job ID 16914735 parsl.executors.status_handling INFO: Allocated block ID 124 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-124 2024-03-08 10:01:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-124 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 67 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 67 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=124 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328cd57021352387d2e04f8d13502849 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f2ca7c97-9e9f-4d63-a6c9-56e441dfc55a_isrForCrosstalkSources_855241_43.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/f2ca7c97-9e9f-4d63-a6c9-56e441dfc55a_isrForCrosstalkSources_855241_43.stderr 2024-03-08 10:01:57 proc_lsst.multi:170 [INFO] [multi] job_id 16914736 proc_lsst.multi INFO: [multi] job_id 16914736 2024-03-08 10:01:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-08 10:01:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914736 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914736 parsl.executors.status_handling DEBUG: Launched block 124 on executor multi with job ID 16914736 parsl.executors.status_handling INFO: Allocated block ID 125 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-125 2024-03-08 10:01:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-125 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 68 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 68 2024-03-08 10:01:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=125 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:01:58 proc_lsst.multi:170 [INFO] [multi] job_id 16914737 parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84f62a5e934a8dde395f3e4e22693eac with result from task 112 proc_lsst.multi INFO: [multi] job_id 16914737 2024-03-08 10:01:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 69 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/19831123-bc35-4b2b-affa-3f4ec146079b_isrForCrosstalkSources_855240_13.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/19831123-bc35-4b2b-affa-3f4ec146079b_isrForCrosstalkSources_855240_13.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 863ce75fd775a8e4b6500b3d43f6f3df with result from task 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-08 10:01:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d81d5922-9f1f-446b-b854-bac768ff0d5f_isrForCrosstalkSources_855155_32.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d81d5922-9f1f-446b-b854-bac768ff0d5f_isrForCrosstalkSources_855155_32.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97faa5e1c80fc50bd1ad506291232c06 with result from task 111 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914737 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b06c0a6e-76c7-4545-8d29-bfb651c1c6a5_isrForCrosstalkSources_855241_27.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b06c0a6e-76c7-4545-8d29-bfb651c1c6a5_isrForCrosstalkSources_855241_27.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5615bcea4d20eabf08551f982e83773a with result from task 103 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914737 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.executors.status_handling DEBUG: Launched block 125 on executor multi with job ID 16914737 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.executors.status_handling INFO: Allocated block ID 126 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/64b5cfca-7b34-4cb3-9226-9133bf329b38_isrForCrosstalkSources_855240_21.stdout parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-126 2024-03-08 10:01:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/64b5cfca-7b34-4cb3-9226-9133bf329b38_isrForCrosstalkSources_855240_21.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f910909c8d653010de3b4ee82c664ab0 with result from task 100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-126 2024-03-08 10:01:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6eba6b60-26d8-4d54-98b4-1b364194be27_isrForCrosstalkSources_855240_11.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6eba6b60-26d8-4d54-98b4-1b364194be27_isrForCrosstalkSources_855240_11.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09390f9e9d2647cba85fa5cc01669617 with result from task 105 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/66b414b3-6ba4-4f8a-8e67-d29d6c8fc481_isrForCrosstalkSources_855235_4.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/66b414b3-6ba4-4f8a-8e67-d29d6c8fc481_isrForCrosstalkSources_855235_4.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3365b2c6fe9ccfc9a38a0e809a5fc79c with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/54c19af1-5e66-4756-a9ce-68d5cd874e14_isrForCrosstalkSources_855240_51.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/54c19af1-5e66-4756-a9ce-68d5cd874e14_isrForCrosstalkSources_855240_51.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69cb09e63c26b876532ad360adb4ceaf with result from task 291 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bee72959-208d-4df2-a827-aa63266e4ac3_isrForCrosstalkSources_855240_45.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/bee72959-208d-4df2-a827-aa63266e4ac3_isrForCrosstalkSources_855240_45.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a224f8861b87eeefb1293a77eae159 with result from task 253 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:01:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a40ee240-0e29-4e41-aa73-b0ad31ec57f3_isrForCrosstalkSources_855242_12.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a40ee240-0e29-4e41-aa73-b0ad31ec57f3_isrForCrosstalkSources_855242_12.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81509b18190e5789d4cdd2861aa96dc6 with result from task 255 proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/5ff50ad6-3924-4f1b-90c1-3d84af93e13b_isrForCrosstalkSources_855231_33.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/5ff50ad6-3924-4f1b-90c1-3d84af93e13b_isrForCrosstalkSources_855231_33.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d243587047603af0dcca70820a59f45 with result from task 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 parsl.dataflow.dflow DEBUG: Task 116 checkpointed proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:01:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5eea1dd5-ad36-47ea-b4be-b2657365e99d_isrForCrosstalkSources_855238_47.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5eea1dd5-ad36-47ea-b4be-b2657365e99d_isrForCrosstalkSources_855238_47.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 223fd19802ebf9005c1697cf72525ff4 with result from task 99 proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:01:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fed7fff3-7507-4eea-8c0a-ae5726dea14f_isrForCrosstalkSources_855238_2.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fed7fff3-7507-4eea-8c0a-ae5726dea14f_isrForCrosstalkSources_855238_2.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ba2b9382a71914ec49587c124ba9689 with result from task 113 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:01:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d5024958-a02a-430b-9ce5-616f1fb5406c_isrForCrosstalkSources_855238_49.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d5024958-a02a-430b-9ce5-616f1fb5406c_isrForCrosstalkSources_855238_49.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d54ac1581d63535a2717fd4d18ef95e with result from task 118 proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d6619e73-8390-48e1-9d85-1b956b8791ca_isrForCrosstalkSources_855156_9.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d6619e73-8390-48e1-9d85-1b956b8791ca_isrForCrosstalkSources_855156_9.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16589d2b7c45418767718ca75c2383de with result from task 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:01:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/de2ae44f-e527-4548-b701-4c8e25f6e66e_isrForCrosstalkSources_855231_5.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/de2ae44f-e527-4548-b701-4c8e25f6e66e_isrForCrosstalkSources_855231_5.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 847d535dab16c377162f1576b40e16df with result from task 308 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:01:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e59026cd-2627-467e-a74e-0dffcb9a4e7b_isrForCrosstalkSources_855231_22.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e59026cd-2627-467e-a74e-0dffcb9a4e7b_isrForCrosstalkSources_855231_22.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2f02f1c16358a128de8e437b1801fb0 with result from task 104 proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:01:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 69 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/dc5e1853-6761-4713-ada1-7ac3ad018019_isrForCrosstalkSources_855155_9.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/dc5e1853-6761-4713-ada1-7ac3ad018019_isrForCrosstalkSources_855155_9.stderr proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 69 2024-03-08 10:01:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:01:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=126 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f963bcf99f918dcb3aa136e98c3d5500 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1aded576-8dd2-4d1a-aaad-18d23df55154_isrForCrosstalkSources_855239_34.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1aded576-8dd2-4d1a-aaad-18d23df55154_isrForCrosstalkSources_855239_34.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeac05c01580cd5ac4e6fc1ef666ff02 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b9148068-2fd2-4b46-b000-eea00b5dfe7d_isrForCrosstalkSources_855154_5.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b9148068-2fd2-4b46-b000-eea00b5dfe7d_isrForCrosstalkSources_855154_5.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16e379e52d876e30c783053c715125da with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/81634672-243f-4bd8-b494-316f94c361fb_isrForCrosstalkSources_855235_12.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/81634672-243f-4bd8-b494-316f94c361fb_isrForCrosstalkSources_855235_12.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0be855e45b5b55d052c00586ccde49f with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/9f120a28-32ce-4c7d-a1eb-cf9f69aa14db_isrForCrosstalkSources_855234_12.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/9f120a28-32ce-4c7d-a1eb-cf9f69aa14db_isrForCrosstalkSources_855234_12.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10674cc85e75b572584898a9e1eabcf5 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7be21375-0635-48b2-8f4d-e3f74ea65359_isrForCrosstalkSources_855236_43.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7be21375-0635-48b2-8f4d-e3f74ea65359_isrForCrosstalkSources_855236_43.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b3c238d405898c5f0f62e662a180453 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/381da02e-6142-40f2-bb70-52d67030094d_isrForCrosstalkSources_855236_58.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/381da02e-6142-40f2-bb70-52d67030094d_isrForCrosstalkSources_855236_58.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 370a0813b622554d2a3a46c7170105f9 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/68507f1d-ea1d-4cf8-8e73-a49652750aa8_isrForCrosstalkSources_855240_39.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/68507f1d-ea1d-4cf8-8e73-a49652750aa8_isrForCrosstalkSources_855240_39.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5c2b03b3cbe0de39eb737af84d762eb with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/de8d3c08-fe38-4d4d-b267-4232b2c99632_isrForCrosstalkSources_855239_17.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/de8d3c08-fe38-4d4d-b267-4232b2c99632_isrForCrosstalkSources_855239_17.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b6b215b14c0ab18b7273cee45df2b2e with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9c9ac09a-5851-4bea-a05f-76918a177984_isrForCrosstalkSources_855232_57.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9c9ac09a-5851-4bea-a05f-76918a177984_isrForCrosstalkSources_855232_57.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca04508a7448ba76dfc1cfc729cf147 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3c03354c-39ae-444a-aaa1-5b3353edce8e_isrForCrosstalkSources_855233_36.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3c03354c-39ae-444a-aaa1-5b3353edce8e_isrForCrosstalkSources_855233_36.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f870cf909bf20b9b94810010d2f34cf5 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/25fc0f29-ecf3-46e2-b7f6-f6616b6f21cb_isrForCrosstalkSources_855242_35.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/25fc0f29-ecf3-46e2-b7f6-f6616b6f21cb_isrForCrosstalkSources_855242_35.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9b3139130602fc20348e87f621d5bb9 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7b2306aa-9475-4a20-b2cc-20bd629cd807_isrForCrosstalkSources_855154_59.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7b2306aa-9475-4a20-b2cc-20bd629cd807_isrForCrosstalkSources_855154_59.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e82a57e92151524fb55c9bf4ec72ce7d with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/234e4b0d-afef-4578-ad92-085896aede53_isrForCrosstalkSources_855238_56.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/234e4b0d-afef-4578-ad92-085896aede53_isrForCrosstalkSources_855238_56.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d46435b05916c2bf05e6c70163f01eb with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/e5c5eb98-5a73-4619-a7c4-1129f74c3801_isrForCrosstalkSources_855240_58.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/e5c5eb98-5a73-4619-a7c4-1129f74c3801_isrForCrosstalkSources_855240_58.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12af1fffec0658ed9c169d437ba0a5a5 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2bbc63f3-0a59-4889-9a28-723db80cd4a0_isrForCrosstalkSources_855238_62.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2bbc63f3-0a59-4889-9a28-723db80cd4a0_isrForCrosstalkSources_855238_62.stderr 2024-03-08 10:02:00 proc_lsst.multi:170 [INFO] [multi] job_id 16914738 proc_lsst.multi INFO: [multi] job_id 16914738 2024-03-08 10:02:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-08 10:02:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914738 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914738 parsl.executors.status_handling DEBUG: Launched block 126 on executor multi with job ID 16914738 parsl.executors.status_handling INFO: Allocated block ID 127 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-127 2024-03-08 10:02:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-127 2024-03-08 10:02:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 70 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 70 2024-03-08 10:02:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=127 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06533f6c90fe397190d40f674ce29bab with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2fe6392a-63b3-466f-bec7-3409b9885ec7_isrForCrosstalkSources_855238_43.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/2fe6392a-63b3-466f-bec7-3409b9885ec7_isrForCrosstalkSources_855238_43.stderr 2024-03-08 10:02:01 proc_lsst.multi:170 [INFO] [multi] job_id 16914739 proc_lsst.multi INFO: [multi] job_id 16914739 2024-03-08 10:02:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-08 10:02:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914739 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914739 parsl.executors.status_handling DEBUG: Launched block 127 on executor multi with job ID 16914739 parsl.executors.status_handling INFO: Allocated block ID 128 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-128 2024-03-08 10:02:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-128 2024-03-08 10:02:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 71 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 71 2024-03-08 10:02:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=128 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:02 proc_lsst.multi:170 [INFO] [multi] job_id 16914740 proc_lsst.multi INFO: [multi] job_id 16914740 2024-03-08 10:02:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-08 10:02:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914740 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914740 parsl.executors.status_handling DEBUG: Launched block 128 on executor multi with job ID 16914740 parsl.executors.status_handling INFO: Allocated block ID 129 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-129 2024-03-08 10:02:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-129 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 72 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 72 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=129 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:02 proc_lsst.multi:170 [INFO] [multi] job_id 16914741 proc_lsst.multi INFO: [multi] job_id 16914741 2024-03-08 10:02:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-08 10:02:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914741 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914741 parsl.executors.status_handling DEBUG: Launched block 129 on executor multi with job ID 16914741 parsl.executors.status_handling INFO: Allocated block ID 130 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-130 2024-03-08 10:02:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-130 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 73 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 73 2024-03-08 10:02:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=130 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a99c5e65640fd75ddbf6cdca661031c9 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d9134325-c22d-4382-b87f-2f3f998bcddf_isrForCrosstalkSources_855155_59.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d9134325-c22d-4382-b87f-2f3f998bcddf_isrForCrosstalkSources_855155_59.stderr 2024-03-08 10:02:03 proc_lsst.multi:170 [INFO] [multi] job_id 16914742 proc_lsst.multi INFO: [multi] job_id 16914742 2024-03-08 10:02:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-08 10:02:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914742 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914742 parsl.executors.status_handling DEBUG: Launched block 130 on executor multi with job ID 16914742 parsl.executors.status_handling INFO: Allocated block ID 131 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-131 2024-03-08 10:02:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-131 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 74 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 74 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=131 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:03 proc_lsst.multi:170 [INFO] [multi] job_id 16914743 proc_lsst.multi INFO: [multi] job_id 16914743 2024-03-08 10:02:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-08 10:02:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914743 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914743 parsl.executors.status_handling DEBUG: Launched block 131 on executor multi with job ID 16914743 parsl.executors.status_handling INFO: Allocated block ID 132 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-132 2024-03-08 10:02:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-132 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 75 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 75 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=132 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd9437d00724aac25478132f3274dc6 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ff32a981-1609-4ab4-a5e9-6237d1493d22_isrForCrosstalkSources_855235_5.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ff32a981-1609-4ab4-a5e9-6237d1493d22_isrForCrosstalkSources_855235_5.stderr 2024-03-08 10:02:03 proc_lsst.multi:170 [INFO] [multi] job_id 16914744 proc_lsst.multi INFO: [multi] job_id 16914744 2024-03-08 10:02:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-08 10:02:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914744 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914744 parsl.executors.status_handling DEBUG: Launched block 132 on executor multi with job ID 16914744 parsl.executors.status_handling INFO: Allocated block ID 133 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-133 2024-03-08 10:02:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-133 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 76 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 76 2024-03-08 10:02:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=133 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:04 proc_lsst.multi:170 [INFO] [multi] job_id 16914745 proc_lsst.multi INFO: [multi] job_id 16914745 2024-03-08 10:02:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-08 10:02:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914745 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914745 parsl.executors.status_handling DEBUG: Launched block 133 on executor multi with job ID 16914745 parsl.executors.status_handling INFO: Allocated block ID 134 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-134 2024-03-08 10:02:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-134 2024-03-08 10:02:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45e9de9ad7616d3ab3ca91089737580 with result from task 344 proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 77 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 77 2024-03-08 10:02:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=134 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/85446f4d-8427-406e-a0ad-81c92fe06f50_isrForCrosstalkSources_855236_2.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/85446f4d-8427-406e-a0ad-81c92fe06f50_isrForCrosstalkSources_855236_2.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e22e1bce626c87504bfd1b46728d28 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5919188b-184e-4705-ab75-9dda660af2a1_isrForCrosstalkSources_855242_18.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5919188b-184e-4705-ab75-9dda660af2a1_isrForCrosstalkSources_855242_18.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6e6dc9ae01212d0057c92f4e88fc068 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c32d7ea4-1956-4210-b978-e0d6a6049a15_isrForCrosstalkSources_855240_42.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c32d7ea4-1956-4210-b978-e0d6a6049a15_isrForCrosstalkSources_855240_42.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e9b90fd7c91fdac71cf2bf4632bad9e with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6b1b8f70-8db1-4216-bdfb-f83b67705a1c_isrForCrosstalkSources_855231_39.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6b1b8f70-8db1-4216-bdfb-f83b67705a1c_isrForCrosstalkSources_855231_39.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bdc55f5d1434da9ffb4853cdfb97d3b with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aeb658cd-3ce3-4075-91f1-a7555ac06717_isrForCrosstalkSources_855242_7.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aeb658cd-3ce3-4075-91f1-a7555ac06717_isrForCrosstalkSources_855242_7.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dab14cf87ede0e6aa396e1fdc493f04 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1c2933c1-f47a-4745-a56f-e390009ca0bc_isrForCrosstalkSources_855232_26.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1c2933c1-f47a-4745-a56f-e390009ca0bc_isrForCrosstalkSources_855232_26.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8991a9d3a56e71bea11f2c00c312efa with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5240718c-40b2-4337-86f6-97cd49b4e934_isrForCrosstalkSources_855235_16.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5240718c-40b2-4337-86f6-97cd49b4e934_isrForCrosstalkSources_855235_16.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d971c90ee94a43ef5ff71f0033f56598 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a6ce6229-6cb8-4407-97bc-fdec02fb9a9f_isrForCrosstalkSources_855232_44.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a6ce6229-6cb8-4407-97bc-fdec02fb9a9f_isrForCrosstalkSources_855232_44.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 719fd30ecb394d234cd416cd45fc3fe0 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3d4441f7-aa51-4bbd-9266-284899365cf5_isrForCrosstalkSources_855231_51.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3d4441f7-aa51-4bbd-9266-284899365cf5_isrForCrosstalkSources_855231_51.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94e750e5888ad5a1dd49ec3fafb5cdc5 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c7b61f91-942b-47cb-9b6b-272ab2f65464_isrForCrosstalkSources_855156_21.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c7b61f91-942b-47cb-9b6b-272ab2f65464_isrForCrosstalkSources_855156_21.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbb53c65ee646af28c61e1a8aab8c557 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/086d9e08-b07c-435d-990c-11471bf1c008_isrForCrosstalkSources_855154_20.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/086d9e08-b07c-435d-990c-11471bf1c008_isrForCrosstalkSources_855154_20.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88f5e94c541bca6cf4b059571c796794 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e8e70a47-bb8c-4b0a-af95-18be3dfc7a67_isrForCrosstalkSources_855235_43.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e8e70a47-bb8c-4b0a-af95-18be3dfc7a67_isrForCrosstalkSources_855235_43.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaf4fac547333b3f5b464e995e1d6825 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/50bc7779-1d15-4441-ba70-e7b605c5d0e6_isrForCrosstalkSources_855231_45.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/50bc7779-1d15-4441-ba70-e7b605c5d0e6_isrForCrosstalkSources_855231_45.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a4b1233da79b0e2295e367ee0e8285b with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9e02a26-b670-4989-a289-635ff9fece81_isrForCrosstalkSources_855156_7.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9e02a26-b670-4989-a289-635ff9fece81_isrForCrosstalkSources_855156_7.stderr 2024-03-08 10:02:07 proc_lsst.multi:170 [INFO] [multi] job_id 16914746 proc_lsst.multi INFO: [multi] job_id 16914746 2024-03-08 10:02:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-08 10:02:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:07 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914746 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914746 parsl.executors.status_handling DEBUG: Launched block 134 on executor multi with job ID 16914746 parsl.executors.status_handling INFO: Allocated block ID 135 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-135 2024-03-08 10:02:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-135 2024-03-08 10:02:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 278f8b0b36bc76543db74c1f9345eef1 with result from task 359 proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:07 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 78 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 78 2024-03-08 10:02:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=135 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6c1a158c-2a8b-45f4-a47c-3adc3a675b3b_isrForCrosstalkSources_855156_41.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6c1a158c-2a8b-45f4-a47c-3adc3a675b3b_isrForCrosstalkSources_855156_41.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2206ad3ecf85a9f524fe080520697f2 with result from task 346 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a317ca89-f8d7-404d-99b7-678a02323853_isrForCrosstalkSources_855235_14.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a317ca89-f8d7-404d-99b7-678a02323853_isrForCrosstalkSources_855235_14.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a59088a8ebaee588294cfb2ab6706d67 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68008abe-4a16-4a89-8bdd-3ac54438d68f_isrForCrosstalkSources_855238_38.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68008abe-4a16-4a89-8bdd-3ac54438d68f_isrForCrosstalkSources_855238_38.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eff3b19ff0d9d1aefc71cec2d5f2be4a with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6665bbe7-a806-44ec-ac35-b8593ebe1376_isrForCrosstalkSources_855241_35.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/6665bbe7-a806-44ec-ac35-b8593ebe1376_isrForCrosstalkSources_855241_35.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b872999d4357917f64ded46acfb076b5 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d58930d-78d4-4518-8f45-11536c8b421a_isrForCrosstalkSources_855154_18.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d58930d-78d4-4518-8f45-11536c8b421a_isrForCrosstalkSources_855154_18.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45bfb7e9aabd44a8b0a576145712c08d with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c114dfd3-52eb-488e-880c-488a8ecf4ea4_isrForCrosstalkSources_855155_25.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/c114dfd3-52eb-488e-880c-488a8ecf4ea4_isrForCrosstalkSources_855155_25.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c19277551bb609454495f9631e8178d with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/92d1ecb2-78ef-4260-ab51-de961802480e_isrForCrosstalkSources_855155_8.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/92d1ecb2-78ef-4260-ab51-de961802480e_isrForCrosstalkSources_855155_8.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 467ff2b0e32f7f734820f6055fc0c7e0 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56ede5e0-66e9-4ba0-a89c-ff09e3867dcc_isrForCrosstalkSources_855237_54.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56ede5e0-66e9-4ba0-a89c-ff09e3867dcc_isrForCrosstalkSources_855237_54.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead730dfcb294165e4be35dc20723e53 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/90db3adb-5900-4d1e-ba10-2c80e15601e8_isrForCrosstalkSources_855233_12.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/90db3adb-5900-4d1e-ba10-2c80e15601e8_isrForCrosstalkSources_855233_12.stderr 2024-03-08 10:02:08 proc_lsst.multi:170 [INFO] [multi] job_id 16914756 proc_lsst.multi INFO: [multi] job_id 16914756 2024-03-08 10:02:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-08 10:02:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:08 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914756 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914756 parsl.executors.status_handling DEBUG: Launched block 135 on executor multi with job ID 16914756 parsl.executors.status_handling INFO: Allocated block ID 136 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-136 2024-03-08 10:02:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-136 2024-03-08 10:02:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:08 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:08 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:08 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 79 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 79 2024-03-08 10:02:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=136 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dcbfde080634256afb8f7d8904c8a78 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/25cc7374-c3a0-4a72-b951-92bf3f69f792_isrForCrosstalkSources_855238_52.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/25cc7374-c3a0-4a72-b951-92bf3f69f792_isrForCrosstalkSources_855238_52.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0238ce9bfe0f38ba7d6347075f837e15 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7eda13b7-b0c7-408b-87b8-c7d8007c667e_isrForCrosstalkSources_855234_29.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/7eda13b7-b0c7-408b-87b8-c7d8007c667e_isrForCrosstalkSources_855234_29.stderr 2024-03-08 10:02:09 proc_lsst.multi:170 [INFO] [multi] job_id 16914757 proc_lsst.multi INFO: [multi] job_id 16914757 2024-03-08 10:02:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-08 10:02:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914757 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914757 parsl.executors.status_handling DEBUG: Launched block 136 on executor multi with job ID 16914757 parsl.executors.status_handling INFO: Allocated block ID 137 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-137 2024-03-08 10:02:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-137 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 80 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 80 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=137 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5dee9a5bd9fd860827b04be2096647a with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b96b4bb4-eba5-443b-85b1-d1dc296d014c_isrForCrosstalkSources_855242_21.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b96b4bb4-eba5-443b-85b1-d1dc296d014c_isrForCrosstalkSources_855242_21.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff64234e76ab1e9acb65b245b9b642de with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3412af94-a3de-47ff-a982-ecc0925b2b46_isrForCrosstalkSources_855237_20.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3412af94-a3de-47ff-a982-ecc0925b2b46_isrForCrosstalkSources_855237_20.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b2501b9046c51c4060321baca1de315 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8ce4a5af-ff8d-4032-8b64-289cd14ed975_isrForCrosstalkSources_855154_48.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8ce4a5af-ff8d-4032-8b64-289cd14ed975_isrForCrosstalkSources_855154_48.stderr 2024-03-08 10:02:09 proc_lsst.multi:170 [INFO] [multi] job_id 16914758 proc_lsst.multi INFO: [multi] job_id 16914758 2024-03-08 10:02:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-08 10:02:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:09 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914758 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914758 parsl.executors.status_handling DEBUG: Launched block 137 on executor multi with job ID 16914758 parsl.executors.status_handling INFO: Allocated block ID 138 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-138 2024-03-08 10:02:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-138 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:09 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 81 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 81 2024-03-08 10:02:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=138 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e728b1f334049fd4910c1abe13fca5d8 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/64c691ea-70dc-4dae-b2e8-356f82f7dd7f_isrForCrosstalkSources_855232_47.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/64c691ea-70dc-4dae-b2e8-356f82f7dd7f_isrForCrosstalkSources_855232_47.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 219654c64c3eb23d6d299cadb06aea8a with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ffe8c9f2-f654-457e-865e-980d127596fe_isrForCrosstalkSources_855236_62.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ffe8c9f2-f654-457e-865e-980d127596fe_isrForCrosstalkSources_855236_62.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18630b15d2ef98eebae7beb7bc5cc953 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aa4f64b4-cae4-42be-b0fe-323a1238d31d_isrForCrosstalkSources_855233_45.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/aa4f64b4-cae4-42be-b0fe-323a1238d31d_isrForCrosstalkSources_855233_45.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3395db3c426319586736fa87d7491859 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c23c9313-ed9f-4157-ad66-4ca027c9c65c_isrForCrosstalkSources_855231_52.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c23c9313-ed9f-4157-ad66-4ca027c9c65c_isrForCrosstalkSources_855231_52.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e460606c90d20d82e9b32118c095c5b with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c655eac8-63d2-4ee7-80de-b2c46ef9b9dd_isrForCrosstalkSources_855239_55.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c655eac8-63d2-4ee7-80de-b2c46ef9b9dd_isrForCrosstalkSources_855239_55.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b64b53b64878290daaa899ab0c076cc with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b0a040ee-f483-4867-80d8-7e41710c7deb_isrForCrosstalkSources_855231_12.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b0a040ee-f483-4867-80d8-7e41710c7deb_isrForCrosstalkSources_855231_12.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3415538ecd5c08a40a169089d81b71cf with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d273d11b-fde1-45f4-b065-d17ab0274162_isrForCrosstalkSources_855240_18.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d273d11b-fde1-45f4-b065-d17ab0274162_isrForCrosstalkSources_855240_18.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 076710caab03735490631e3ea402b4f1 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2dc61eaf-5fd7-4a8a-9133-f045ba88bcda_isrForCrosstalkSources_855231_20.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2dc61eaf-5fd7-4a8a-9133-f045ba88bcda_isrForCrosstalkSources_855231_20.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d8e18fd72b6de4fa722090a2e364bf with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/85370ba3-60a8-47fc-819c-506a13ca2941_isrForCrosstalkSources_855156_45.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/85370ba3-60a8-47fc-819c-506a13ca2941_isrForCrosstalkSources_855156_45.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81f5013044c566202ced590d1f2353d0 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b04370a6-6179-4349-81be-8413cc340ecc_isrForCrosstalkSources_855234_2.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b04370a6-6179-4349-81be-8413cc340ecc_isrForCrosstalkSources_855234_2.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52a24f2de64068fb74d3a44545b301dd with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a88de9e-c240-4bc8-bee8-e112e6765cbc_isrForCrosstalkSources_855235_31.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a88de9e-c240-4bc8-bee8-e112e6765cbc_isrForCrosstalkSources_855235_31.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f33b855f8ed61185c7f3664ecc56df4 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5207160d-3701-4096-ae21-00818136e64b_isrForCrosstalkSources_855233_44.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5207160d-3701-4096-ae21-00818136e64b_isrForCrosstalkSources_855233_44.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10fd002830f6ff2f2821cf855de22e8b with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f1033c6f-8546-46b4-bdb9-2edab5da9d28_isrForCrosstalkSources_855156_60.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f1033c6f-8546-46b4-bdb9-2edab5da9d28_isrForCrosstalkSources_855156_60.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 696d53ea1e07e656ca2ec97265a6dd91 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ce1cb304-0bbf-4d8a-b779-07b18fc9a6cf_isrForCrosstalkSources_855242_49.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ce1cb304-0bbf-4d8a-b779-07b18fc9a6cf_isrForCrosstalkSources_855242_49.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 357c4f13adbe2d48e71dad884268008e with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/92cbe666-a7b1-4e84-8132-1cc4f7b54f0e_isrForCrosstalkSources_855154_51.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/92cbe666-a7b1-4e84-8132-1cc4f7b54f0e_isrForCrosstalkSources_855154_51.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9e4770e3a8f3908ecfd01812d9c8ebe with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68d930e7-e624-4249-89b7-ba9cb33c60eb_isrForCrosstalkSources_855238_4.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/68d930e7-e624-4249-89b7-ba9cb33c60eb_isrForCrosstalkSources_855238_4.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2d5e13c9b6055d0a20c102137f16c70 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ebf80c40-0c48-417f-ad37-73b2838edf1c_isrForCrosstalkSources_855236_34.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ebf80c40-0c48-417f-ad37-73b2838edf1c_isrForCrosstalkSources_855236_34.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf4aad9c6de4b3e5adef6f1bc875ff0c with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2411d2e9-e4cf-4293-a98a-8b486b4bee9f_isrForCrosstalkSources_855231_14.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/2411d2e9-e4cf-4293-a98a-8b486b4bee9f_isrForCrosstalkSources_855231_14.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c9dd2161ff5fa0fd4f6c126d85770cf with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6ac4c800-b0f9-4d7d-a85d-21510240e70d_isrForCrosstalkSources_855242_17.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6ac4c800-b0f9-4d7d-a85d-21510240e70d_isrForCrosstalkSources_855242_17.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b29b02151e509f7672629bcf97883551 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/422ee4ea-fbf0-4123-8c57-4d8b4b3b7a1e_isrForCrosstalkSources_855233_26.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/422ee4ea-fbf0-4123-8c57-4d8b4b3b7a1e_isrForCrosstalkSources_855233_26.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752834fc5bcf0d1ab7fbad64dbe65965 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/69e55859-64d0-41f4-a005-e7f28fa2839a_isrForCrosstalkSources_855232_10.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/69e55859-64d0-41f4-a005-e7f28fa2839a_isrForCrosstalkSources_855232_10.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1f5d602e99d7a2fcde84020fe2600f with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fa7c5d43-af3e-4f51-8e09-caedc403b86a_isrForCrosstalkSources_855239_33.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fa7c5d43-af3e-4f51-8e09-caedc403b86a_isrForCrosstalkSources_855239_33.stderr 2024-03-08 10:02:13 proc_lsst.multi:170 [INFO] [multi] job_id 16914759 proc_lsst.multi INFO: [multi] job_id 16914759 2024-03-08 10:02:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-08 10:02:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914759 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914759 parsl.executors.status_handling DEBUG: Launched block 138 on executor multi with job ID 16914759 parsl.executors.status_handling INFO: Allocated block ID 139 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-139 2024-03-08 10:02:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-139 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 82 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 82 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=139 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da090dc54e2786a2fe0f010d45905424 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/02c18129-53e5-4ac5-b664-3acf00977d27_isrForCrosstalkSources_855155_51.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/02c18129-53e5-4ac5-b664-3acf00977d27_isrForCrosstalkSources_855155_51.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d438a8c545a8a96de68aa4f0c34217d6 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c1d01ccf-78fc-4796-bfaf-d47b36244442_isrForCrosstalkSources_855232_12.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c1d01ccf-78fc-4796-bfaf-d47b36244442_isrForCrosstalkSources_855232_12.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d620abff0cab20d23f0ccc32089cafaf with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/816fe5b0-4e10-4dfc-8452-3b5b05f8fb34_isrForCrosstalkSources_855237_56.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/816fe5b0-4e10-4dfc-8452-3b5b05f8fb34_isrForCrosstalkSources_855237_56.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 961e25c0df547d36325883bdd5d9e826 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d88cf86-881d-4fe1-8e9b-f11c1b7f4c06_isrForCrosstalkSources_855237_24.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d88cf86-881d-4fe1-8e9b-f11c1b7f4c06_isrForCrosstalkSources_855237_24.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ed1f6ad26ed71cbafca504479d59c72 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b7d1277e-67b5-4673-bcce-5f7808676f7f_isrForCrosstalkSources_855232_6.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b7d1277e-67b5-4673-bcce-5f7808676f7f_isrForCrosstalkSources_855232_6.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b3b50bbb5e92427436ca2753d8effb6 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f7e62045-4e19-42fd-92b0-ddb8fa801b2d_isrForCrosstalkSources_855235_18.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f7e62045-4e19-42fd-92b0-ddb8fa801b2d_isrForCrosstalkSources_855235_18.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d96c98f798ada4eae7f9992eb8d2ec8b with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e6a22cc3-beaa-406a-a9ec-e9c217c4479d_isrForCrosstalkSources_855232_11.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e6a22cc3-beaa-406a-a9ec-e9c217c4479d_isrForCrosstalkSources_855232_11.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf5ef5d08b068f077c3c974e8569365 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c095cdff-2d34-411a-801e-53dc01a02236_isrForCrosstalkSources_855239_23.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c095cdff-2d34-411a-801e-53dc01a02236_isrForCrosstalkSources_855239_23.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f547f43cf5f81f2b6b1f0351ab1ed1 with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/54e5f70f-7e67-47b3-b726-cbfde1b4e59b_isrForCrosstalkSources_855156_18.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/54e5f70f-7e67-47b3-b726-cbfde1b4e59b_isrForCrosstalkSources_855156_18.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17ea4d7cce89a0afd4b46d5a8dccbcce with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20c76607-16b8-4bab-a8fc-eaa1bec8d5d0_isrForCrosstalkSources_855156_1.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20c76607-16b8-4bab-a8fc-eaa1bec8d5d0_isrForCrosstalkSources_855156_1.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09331432f0312b98939036d94adb885d with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a4d8063-b2af-42a9-a3f6-c6ae99e555f6_isrForCrosstalkSources_855156_36.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a4d8063-b2af-42a9-a3f6-c6ae99e555f6_isrForCrosstalkSources_855156_36.stderr 2024-03-08 10:02:13 proc_lsst.multi:170 [INFO] [multi] job_id 16914760 proc_lsst.multi INFO: [multi] job_id 16914760 2024-03-08 10:02:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-08 10:02:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:13 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914760 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914760 parsl.executors.status_handling DEBUG: Launched block 139 on executor multi with job ID 16914760 parsl.executors.status_handling INFO: Allocated block ID 140 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-140 2024-03-08 10:02:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-140 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:13 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 83 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 83 2024-03-08 10:02:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=140 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec9f02fbbdcfe8d04278d3d6305c98b2 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9831555a-bf28-4bc0-a478-cc235dd07cea_isrForCrosstalkSources_855239_35.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9831555a-bf28-4bc0-a478-cc235dd07cea_isrForCrosstalkSources_855239_35.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7afd6b2bc22087f8bda14eca143f4157 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a11b9ecb-d230-46f5-a943-c829d0aa594a_isrForCrosstalkSources_855235_61.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a11b9ecb-d230-46f5-a943-c829d0aa594a_isrForCrosstalkSources_855235_61.stderr 2024-03-08 10:02:14 proc_lsst.multi:170 [INFO] [multi] job_id 16914761 proc_lsst.multi INFO: [multi] job_id 16914761 2024-03-08 10:02:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-08 10:02:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914761 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914761 parsl.executors.status_handling DEBUG: Launched block 140 on executor multi with job ID 16914761 parsl.executors.status_handling INFO: Allocated block ID 141 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-141 2024-03-08 10:02:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-141 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 84 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 84 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=141 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 180a7eb145ff04e42b1ce6b7a9995712 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4a89aca6-9180-42bf-abaf-e63f56584d40_isrForCrosstalkSources_855231_2.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4a89aca6-9180-42bf-abaf-e63f56584d40_isrForCrosstalkSources_855231_2.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab89f79e0c2ec4e200a0d85b16aa62b3 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/49db40f7-86d4-46e8-a913-7fa37dc323c4_isrForCrosstalkSources_855234_1.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/49db40f7-86d4-46e8-a913-7fa37dc323c4_isrForCrosstalkSources_855234_1.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffe8bcf30c81554ff954ae2f627648a with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2bc2ef5a-3f87-4529-a2b4-bdd41252b1c2_isrForCrosstalkSources_855240_31.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2bc2ef5a-3f87-4529-a2b4-bdd41252b1c2_isrForCrosstalkSources_855240_31.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca53068b33dda7054cc23f29da9a2284 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/54e57280-85d5-471e-aa13-5f430f284210_isrForCrosstalkSources_855155_1.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/54e57280-85d5-471e-aa13-5f430f284210_isrForCrosstalkSources_855155_1.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4f9c3ad9b5fc72db8697d73e354a322 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fc06db14-d9d3-4eab-b56e-55ab67e9a5bf_isrForCrosstalkSources_855238_22.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/fc06db14-d9d3-4eab-b56e-55ab67e9a5bf_isrForCrosstalkSources_855238_22.stderr 2024-03-08 10:02:14 proc_lsst.multi:170 [INFO] [multi] job_id 16914762 proc_lsst.multi INFO: [multi] job_id 16914762 2024-03-08 10:02:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-08 10:02:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:14 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914762 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914762 parsl.executors.status_handling DEBUG: Launched block 141 on executor multi with job ID 16914762 parsl.executors.status_handling INFO: Allocated block ID 142 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-142 2024-03-08 10:02:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-142 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:14 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 85 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 85 2024-03-08 10:02:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=142 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba2583d81d38c169b5ce3f8cc7fd6ea3 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0d27690c-25d3-4b41-be05-493453547140_isrForCrosstalkSources_855239_57.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0d27690c-25d3-4b41-be05-493453547140_isrForCrosstalkSources_855239_57.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b83241f79dd0c6ae19aa6d13e1e16083 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/434dba8d-47ef-4d5e-887e-9f099ca70848_isrForCrosstalkSources_855236_50.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/434dba8d-47ef-4d5e-887e-9f099ca70848_isrForCrosstalkSources_855236_50.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry defe19145a2bad124eee141b1c8ef45a with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b46fe0ab-5a51-4620-9700-b4c9ff89f207_isrForCrosstalkSources_855241_50.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b46fe0ab-5a51-4620-9700-b4c9ff89f207_isrForCrosstalkSources_855241_50.stderr 2024-03-08 10:02:15 proc_lsst.multi:170 [INFO] [multi] job_id 16914763 proc_lsst.multi INFO: [multi] job_id 16914763 2024-03-08 10:02:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-08 10:02:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914763 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914763 parsl.executors.status_handling DEBUG: Launched block 142 on executor multi with job ID 16914763 parsl.executors.status_handling INFO: Allocated block ID 143 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-143 2024-03-08 10:02:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-143 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 86 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 86 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=143 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b49fc10d068f87bef13ecc403fcef63 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1a9b05b0-8a09-49e6-a23a-f95c5f3772b7_isrForCrosstalkSources_855240_36.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1a9b05b0-8a09-49e6-a23a-f95c5f3772b7_isrForCrosstalkSources_855240_36.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06e4e0703ea04c871846aede743c3c5a with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/65b601b8-b34a-492a-a6e3-6437edb1a9f2_isrForCrosstalkSources_855238_44.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/65b601b8-b34a-492a-a6e3-6437edb1a9f2_isrForCrosstalkSources_855238_44.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6e02efecc58a395026440fb44781d13 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8a9b12ac-5d31-43db-a997-e255a0f92467_isrForCrosstalkSources_855239_22.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8a9b12ac-5d31-43db-a997-e255a0f92467_isrForCrosstalkSources_855239_22.stderr 2024-03-08 10:02:15 proc_lsst.multi:170 [INFO] [multi] job_id 16914764 proc_lsst.multi INFO: [multi] job_id 16914764 2024-03-08 10:02:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-08 10:02:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:15 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914764 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914764 parsl.executors.status_handling DEBUG: Launched block 143 on executor multi with job ID 16914764 parsl.executors.status_handling INFO: Allocated block ID 144 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-144 2024-03-08 10:02:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-144 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:15 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 87 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 87 2024-03-08 10:02:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=144 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0a79cc115c60a05151e4b5ff253f3d6 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/49eb0497-d028-4699-9480-ccb504661d07_isrForCrosstalkSources_855237_39.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/49eb0497-d028-4699-9480-ccb504661d07_isrForCrosstalkSources_855237_39.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4dcf1da2025a8065bdfe2944d3a10b9 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/c0cffadf-64b2-41b9-938b-a7f9140ca3ce_isrForCrosstalkSources_855234_40.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/c0cffadf-64b2-41b9-938b-a7f9140ca3ce_isrForCrosstalkSources_855234_40.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1ddb5b27867f65149a10d06103fb718 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/687110ba-6c2c-4a7a-a705-c8fb4bdf05e6_isrForCrosstalkSources_855237_41.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/687110ba-6c2c-4a7a-a705-c8fb4bdf05e6_isrForCrosstalkSources_855237_41.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f410c4e69fc2abc66847fd07665dbf54 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/835a3ed5-153c-423e-8626-ba402dbc37b7_isrForCrosstalkSources_855155_34.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/835a3ed5-153c-423e-8626-ba402dbc37b7_isrForCrosstalkSources_855155_34.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f63f6f2e2c1e8c81ea7490c51f0f47b8 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/376e5909-6ae4-405b-a632-eb6194f066e8_isrForCrosstalkSources_855240_7.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/376e5909-6ae4-405b-a632-eb6194f066e8_isrForCrosstalkSources_855240_7.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf38565c463bf93ac26a6df5f15d269 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/be912ab7-fc8a-4822-9cba-ae8e9bed788e_isrForCrosstalkSources_855232_36.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/be912ab7-fc8a-4822-9cba-ae8e9bed788e_isrForCrosstalkSources_855232_36.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fd5a46521f0805f933afead45181da6 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ef28b32e-57b5-477b-b709-f47ac2cd05ad_isrForCrosstalkSources_855155_47.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ef28b32e-57b5-477b-b709-f47ac2cd05ad_isrForCrosstalkSources_855155_47.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99d6328e05dfa3a77ec057e78826c997 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a3c5444c-cd44-4b6a-8a93-e48122973e55_isrForCrosstalkSources_855234_6.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a3c5444c-cd44-4b6a-8a93-e48122973e55_isrForCrosstalkSources_855234_6.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01652aa8c097711eb2c94d1d8c00bdc9 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0dd0705f-b5e9-4083-9801-85038425b5b3_isrForCrosstalkSources_855239_59.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0dd0705f-b5e9-4083-9801-85038425b5b3_isrForCrosstalkSources_855239_59.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d61aac6f4e95af8cf75fc6b2a3af4d25 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab5336e6-df99-42dd-ad60-8896b734dac2_isrForCrosstalkSources_855235_6.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab5336e6-df99-42dd-ad60-8896b734dac2_isrForCrosstalkSources_855235_6.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 185407fcd36a7d0838ec20cb849e2658 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a401edc5-a2c0-4bd4-a729-b24abf7ffb0e_isrForCrosstalkSources_855237_3.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a401edc5-a2c0-4bd4-a729-b24abf7ffb0e_isrForCrosstalkSources_855237_3.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b1f6effbfb66dfcb1ffe9fa198cf4bf with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/9cd98147-48d7-45c5-be6f-448cad0c4b94_isrForCrosstalkSources_855242_45.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/9cd98147-48d7-45c5-be6f-448cad0c4b94_isrForCrosstalkSources_855242_45.stderr 2024-03-08 10:02:16 proc_lsst.multi:170 [INFO] [multi] job_id 16914765 proc_lsst.multi INFO: [multi] job_id 16914765 2024-03-08 10:02:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-08 10:02:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914765 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914765 parsl.executors.status_handling DEBUG: Launched block 144 on executor multi with job ID 16914765 parsl.executors.status_handling INFO: Allocated block ID 145 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-145 2024-03-08 10:02:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-145 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 88 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 88 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=145 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 045fb612fa2cbdebeb45f96fcd179657 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6b572aec-1b2b-4f31-a09f-31547d21b80b_isrForCrosstalkSources_855233_48.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6b572aec-1b2b-4f31-a09f-31547d21b80b_isrForCrosstalkSources_855233_48.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3ce7986cb61839834e8056313885f55 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c32a9d6f-7a44-445c-8fc8-2478e923c69a_isrForCrosstalkSources_855237_11.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c32a9d6f-7a44-445c-8fc8-2478e923c69a_isrForCrosstalkSources_855237_11.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 416a43b93c7ac0f8905541fe6438c4aa with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d71d90b7-fd6d-416f-9cc1-f83b26f755fb_isrForCrosstalkSources_855237_59.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d71d90b7-fd6d-416f-9cc1-f83b26f755fb_isrForCrosstalkSources_855237_59.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a79e7ffa8ea9e0cc3cabe64a463f35c with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc549441-aec6-4782-b9c8-3e40f098b558_isrForCrosstalkSources_855234_19.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc549441-aec6-4782-b9c8-3e40f098b558_isrForCrosstalkSources_855234_19.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d42456f420f6e4966f64c94e677fb1 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3025eb4c-61e3-4cf4-abf9-8cb73593aa01_isrForCrosstalkSources_855237_36.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/3025eb4c-61e3-4cf4-abf9-8cb73593aa01_isrForCrosstalkSources_855237_36.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9643169e52c4074b4ac101f84ee5ea95 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/dedbc313-eafc-4e6b-bcf9-8439a915f620_isrForCrosstalkSources_855235_41.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/dedbc313-eafc-4e6b-bcf9-8439a915f620_isrForCrosstalkSources_855235_41.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a3a6cecc3bc78f9cb1d2aff6183913 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93626210-e536-46f8-a83a-d389ca3121c4_isrForCrosstalkSources_855155_15.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/93626210-e536-46f8-a83a-d389ca3121c4_isrForCrosstalkSources_855155_15.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88a5d77dcc5b6f7f380e9366b56c8c87 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f548239-fd73-443d-bddd-d1f31898d7b9_isrForCrosstalkSources_855241_19.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f548239-fd73-443d-bddd-d1f31898d7b9_isrForCrosstalkSources_855241_19.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd1297007a360d0329f08ba17fa7ca21 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/194da3ff-4ef9-4461-a9a5-0cc11a3bf855_isrForCrosstalkSources_855156_59.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/194da3ff-4ef9-4461-a9a5-0cc11a3bf855_isrForCrosstalkSources_855156_59.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77ab53ced259861f724b5c195b394c5c with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/38c40461-1175-4a52-89fe-0f51c4003f4f_isrForCrosstalkSources_855232_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/38c40461-1175-4a52-89fe-0f51c4003f4f_isrForCrosstalkSources_855232_17.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 793f6cfec19e1035524fb598bf8e008a with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5ea22d61-ffc0-4c5d-9585-1314433e9e9b_isrForCrosstalkSources_855233_24.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5ea22d61-ffc0-4c5d-9585-1314433e9e9b_isrForCrosstalkSources_855233_24.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a61283117804e1a756c03f2bc704f2e with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/9664a0b8-53f4-422a-a37e-41343a0d7828_isrForCrosstalkSources_855156_58.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/9664a0b8-53f4-422a-a37e-41343a0d7828_isrForCrosstalkSources_855156_58.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59f7bfe25464b6270a6a93fa646d9c2a with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a773ce8b-706e-4e13-93a9-4156df5e08da_isrForCrosstalkSources_855238_20.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a773ce8b-706e-4e13-93a9-4156df5e08da_isrForCrosstalkSources_855238_20.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09f5a1d846eb4a136495afd5e6cbbcbd with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/37795145-9588-4c00-a0bb-127b4d5ee98a_isrForCrosstalkSources_855234_61.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/37795145-9588-4c00-a0bb-127b4d5ee98a_isrForCrosstalkSources_855234_61.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a825e295af3a322394756be9c3efb41 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a5a98d09-e153-41b9-92c5-33108c6473ae_isrForCrosstalkSources_855238_41.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a5a98d09-e153-41b9-92c5-33108c6473ae_isrForCrosstalkSources_855238_41.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78874388f6c3d1413eb8547e4b4f0df2 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/17787604-2e8a-44b4-a209-a82653f3d3fe_isrForCrosstalkSources_855235_44.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/17787604-2e8a-44b4-a209-a82653f3d3fe_isrForCrosstalkSources_855235_44.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b13d596ce7b9455825f401f3bd54323 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/edba68ad-b074-4e9a-a713-86bd87cb1494_isrForCrosstalkSources_855156_46.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/edba68ad-b074-4e9a-a713-86bd87cb1494_isrForCrosstalkSources_855156_46.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62dd01c670b71ad109a010404932e529 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/5c7ba540-d26d-4d44-83d1-412cf7a7876a_isrForCrosstalkSources_855232_41.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/5c7ba540-d26d-4d44-83d1-412cf7a7876a_isrForCrosstalkSources_855232_41.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0639fc79681e9dea23aace6fa60b1490 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2cbeb5d-23c1-4118-a0f8-3c89e100b28f_isrForCrosstalkSources_855239_27.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2cbeb5d-23c1-4118-a0f8-3c89e100b28f_isrForCrosstalkSources_855239_27.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b75649da4479a5ecac14f1df3058e4 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/689479e6-a188-4f34-8e84-04a640f481ba_isrForCrosstalkSources_855242_29.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/689479e6-a188-4f34-8e84-04a640f481ba_isrForCrosstalkSources_855242_29.stderr 2024-03-08 10:02:16 proc_lsst.multi:170 [INFO] [multi] job_id 16914766 proc_lsst.multi INFO: [multi] job_id 16914766 2024-03-08 10:02:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-08 10:02:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914766 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914766 parsl.executors.status_handling DEBUG: Launched block 145 on executor multi with job ID 16914766 parsl.executors.status_handling INFO: Allocated block ID 146 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-146 2024-03-08 10:02:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-146 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 89 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 89 2024-03-08 10:02:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=146 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d234b0b2487b6f5d79beebdbb2bf9bb2 with result from task 245 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1648f305-d415-41f0-9075-537e073ae294_isrForCrosstalkSources_855239_28.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1648f305-d415-41f0-9075-537e073ae294_isrForCrosstalkSources_855239_28.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b69aa071408765008e1b68c3a2087bc2 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b9e58d2a-1521-47f6-9e16-a19fd0dfca06_isrForCrosstalkSources_855234_9.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/b9e58d2a-1521-47f6-9e16-a19fd0dfca06_isrForCrosstalkSources_855234_9.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff3ace6aca7cd7c7416610e3d9b11b35 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/26d09a68-8a4c-49e7-83e8-9058c3679adb_isrForCrosstalkSources_855155_50.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/26d09a68-8a4c-49e7-83e8-9058c3679adb_isrForCrosstalkSources_855155_50.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efa2cfaae6d8e6d346c084ad3764dcfe with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0bb776ef-2aea-4fb9-80c3-af8cc8653456_isrForCrosstalkSources_855237_44.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0bb776ef-2aea-4fb9-80c3-af8cc8653456_isrForCrosstalkSources_855237_44.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0495e649b5955769516180fb61b1e6b1 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/36ba3895-4bb1-4e24-bb2a-a3430259d8f1_isrForCrosstalkSources_855240_30.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/36ba3895-4bb1-4e24-bb2a-a3430259d8f1_isrForCrosstalkSources_855240_30.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f772f9a3705a7283c78b4075024d4f19 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c37b71c6-bc3f-4bb7-8f99-c4512136c437_isrForCrosstalkSources_855231_60.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c37b71c6-bc3f-4bb7-8f99-c4512136c437_isrForCrosstalkSources_855231_60.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beb7a134c54ef91c347cf953cbaa27e4 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a50bee3f-f8fc-4f2c-b9af-a0fa72dbe09a_isrForCrosstalkSources_855239_19.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a50bee3f-f8fc-4f2c-b9af-a0fa72dbe09a_isrForCrosstalkSources_855239_19.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7a88a8794936441324ef33f5a231852 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/80db1a87-9015-4a94-b298-79424e6f7562_isrForCrosstalkSources_855240_37.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/80db1a87-9015-4a94-b298-79424e6f7562_isrForCrosstalkSources_855240_37.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6f5e73bc4824fcb3091db3e71d68c1e with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f686054b-8ca1-46e0-bf91-8f022a9fc2e3_isrForCrosstalkSources_855155_16.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f686054b-8ca1-46e0-bf91-8f022a9fc2e3_isrForCrosstalkSources_855155_16.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bbb06471ad93d0e9085cb9dd14f4c65 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1a06d35a-6dbb-43ca-bf64-963b348b0b5e_isrForCrosstalkSources_855238_28.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1a06d35a-6dbb-43ca-bf64-963b348b0b5e_isrForCrosstalkSources_855238_28.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e77a93d4f402892a80b938b3cf0bceb0 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bb71d1c-9d86-49ba-9d82-06c4191d7be0_isrForCrosstalkSources_855231_57.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bb71d1c-9d86-49ba-9d82-06c4191d7be0_isrForCrosstalkSources_855231_57.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 105b534103c29a35b6659f79b6ece258 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/422ad764-8d2f-4ab5-b7de-aaf5ccfca40a_isrForCrosstalkSources_855156_44.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/422ad764-8d2f-4ab5-b7de-aaf5ccfca40a_isrForCrosstalkSources_855156_44.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d24ab51aec63e80b8613034eea7b02d with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6f0acc6-5ae5-4d29-b597-333beac4d038_isrForCrosstalkSources_855232_53.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6f0acc6-5ae5-4d29-b597-333beac4d038_isrForCrosstalkSources_855232_53.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be898cd7820eac490f076cc3d4587290 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e673ce2d-afa9-427d-9848-15666a5d1d96_isrForCrosstalkSources_855154_30.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e673ce2d-afa9-427d-9848-15666a5d1d96_isrForCrosstalkSources_855154_30.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 299682af4a3850a4bcddd36c4b12e5d9 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc9a2663-7c9d-4f8e-961e-5522236a8507_isrForCrosstalkSources_855235_58.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc9a2663-7c9d-4f8e-961e-5522236a8507_isrForCrosstalkSources_855235_58.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dbd4a3c0be9dc82e7b440ed47b906cd with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/81f09af6-83c3-4c97-9c77-8b7d474cb4e9_isrForCrosstalkSources_855236_60.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/81f09af6-83c3-4c97-9c77-8b7d474cb4e9_isrForCrosstalkSources_855236_60.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 425f38f53a6a0f78e191c39685fed37e with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8124743b-a73b-4abc-93cc-bb9cca132bb9_isrForCrosstalkSources_855239_58.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/8124743b-a73b-4abc-93cc-bb9cca132bb9_isrForCrosstalkSources_855239_58.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af7e20026fd7ab27581cf6063e246e0e with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3eac354a-f8f9-4c89-b456-14c292123719_isrForCrosstalkSources_855241_18.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3eac354a-f8f9-4c89-b456-14c292123719_isrForCrosstalkSources_855241_18.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55aa3941f94f1153d9a2a101f0314bcd with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d7c253b8-b959-433f-8be9-c82a119c216e_isrForCrosstalkSources_855240_20.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d7c253b8-b959-433f-8be9-c82a119c216e_isrForCrosstalkSources_855240_20.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 932f084f07816f813144fd257d686aaf with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2eb0b5c3-6121-4081-a4f5-e4a55b8e9763_isrForCrosstalkSources_855155_58.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2eb0b5c3-6121-4081-a4f5-e4a55b8e9763_isrForCrosstalkSources_855155_58.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6139cf7eae9571a770d15a7b0f7b8ff8 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7f5eb7dc-7200-43cd-9bbe-72954b2e2bd0_isrForCrosstalkSources_855231_35.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7f5eb7dc-7200-43cd-9bbe-72954b2e2bd0_isrForCrosstalkSources_855231_35.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 518570ecc95c1d4f30efbea969132012 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/51ec8ca5-fbf9-4156-a209-1c52564c2790_isrForCrosstalkSources_855242_33.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/51ec8ca5-fbf9-4156-a209-1c52564c2790_isrForCrosstalkSources_855242_33.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf59bac2e23d530ba6ea9b46c1f0349 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6905d954-4a5a-4e42-8c43-c658a2c74823_isrForCrosstalkSources_855231_36.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/6905d954-4a5a-4e42-8c43-c658a2c74823_isrForCrosstalkSources_855231_36.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d581f70f552d1c057a7e7b81b28d9757 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/1c9605d5-cad0-497e-8abb-85d2c260ae64_isrForCrosstalkSources_855235_15.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/1c9605d5-cad0-497e-8abb-85d2c260ae64_isrForCrosstalkSources_855235_15.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4fb0abf65720d5bd3f17a6738018c23 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/5c35f1bb-8614-4025-acf6-f5b51624489e_isrForCrosstalkSources_855240_54.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/5c35f1bb-8614-4025-acf6-f5b51624489e_isrForCrosstalkSources_855240_54.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e44567cec1d754c05712601815b99aa with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/cbac35c0-7518-41e6-8bf5-82600b0822f4_isrForCrosstalkSources_855154_17.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/cbac35c0-7518-41e6-8bf5-82600b0822f4_isrForCrosstalkSources_855154_17.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4de99f4234a00b4769e41b6a317c61ad with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/3c7edd30-8946-4264-bd8a-c5bda121af2b_isrForCrosstalkSources_855154_62.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/3c7edd30-8946-4264-bd8a-c5bda121af2b_isrForCrosstalkSources_855154_62.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b52dc9849b340f444d1ea508a586d05 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/8a048b0a-19f0-42ef-b065-eb26e9997c00_isrForCrosstalkSources_855231_30.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/8a048b0a-19f0-42ef-b065-eb26e9997c00_isrForCrosstalkSources_855231_30.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61b3399a023ef0a606d04e29306428d3 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/9b850888-d5c4-428f-aa2b-3d564316bebd_isrForCrosstalkSources_855235_13.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/9b850888-d5c4-428f-aa2b-3d564316bebd_isrForCrosstalkSources_855235_13.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 260fc24ed270a4e4b498406b27525cac with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd95eb13-4336-4112-b8a3-c4cf6822019b_isrForCrosstalkSources_855154_37.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd95eb13-4336-4112-b8a3-c4cf6822019b_isrForCrosstalkSources_855154_37.stderr 2024-03-08 10:02:17 proc_lsst.multi:170 [INFO] [multi] job_id 16914767 proc_lsst.multi INFO: [multi] job_id 16914767 2024-03-08 10:02:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-08 10:02:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914767 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914767 parsl.executors.status_handling DEBUG: Launched block 146 on executor multi with job ID 16914767 parsl.executors.status_handling INFO: Allocated block ID 147 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-147 2024-03-08 10:02:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-147 2024-03-08 10:02:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 90 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 90 2024-03-08 10:02:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=147 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9d27ed7a617205b3175c715a1dad82 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0a72e735-1aa9-49fc-a5e1-a0695e1d1398_isrForCrosstalkSources_855156_23.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/0a72e735-1aa9-49fc-a5e1-a0695e1d1398_isrForCrosstalkSources_855156_23.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dc664ac691d2f81e8a633aa9b0131cd with result from task 406 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/b357fff7-dc45-47d7-b56f-aaa37f3c847b_isrForCrosstalkSources_855156_16.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/b357fff7-dc45-47d7-b56f-aaa37f3c847b_isrForCrosstalkSources_855156_16.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80141809fb27e8f71b987f95c7abfdc0 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03866d3e-3030-4285-8da8-8ef92b000e3a_isrForCrosstalkSources_855238_46.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03866d3e-3030-4285-8da8-8ef92b000e3a_isrForCrosstalkSources_855238_46.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47be653f2ac4ae4d8bdce502615674a2 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3c5fc83b-4319-426b-b821-bedbb3140815_isrForCrosstalkSources_855234_15.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3c5fc83b-4319-426b-b821-bedbb3140815_isrForCrosstalkSources_855234_15.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 422f55135b5ff8064b08a21937d848ac with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7d8717fe-14b7-477f-b687-0854aa024f64_isrForCrosstalkSources_855154_6.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7d8717fe-14b7-477f-b687-0854aa024f64_isrForCrosstalkSources_855154_6.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffa5e68f08998c71d8008185357329c2 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e7e8965a-60af-4e4e-8d17-2d9d0847978b_isrForCrosstalkSources_855242_48.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e7e8965a-60af-4e4e-8d17-2d9d0847978b_isrForCrosstalkSources_855242_48.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25328a7ee75250c336df1293d1985656 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/e99b6079-1ccf-4600-a72d-1e812416c20e_isrForCrosstalkSources_855236_29.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/e99b6079-1ccf-4600-a72d-1e812416c20e_isrForCrosstalkSources_855236_29.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d93aba22532957b160d7f9e879ac6dd4 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/38f23734-95fa-4565-93d0-b58addd55708_isrForCrosstalkSources_855235_25.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/38f23734-95fa-4565-93d0-b58addd55708_isrForCrosstalkSources_855235_25.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f742d3e398e9830d4ac381a5ff737c5 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c11c7160-a5ff-4b43-a11f-fcf6d024bffc_isrForCrosstalkSources_855156_10.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c11c7160-a5ff-4b43-a11f-fcf6d024bffc_isrForCrosstalkSources_855156_10.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89d07624b62f034239ee6c9453a3625c with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/eafafeb5-2076-405d-8daf-c368d655cf1e_isrForCrosstalkSources_855237_23.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/eafafeb5-2076-405d-8daf-c368d655cf1e_isrForCrosstalkSources_855237_23.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41f8f073ec58a3573b6bb12ff7017f94 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f9b2da64-974e-490e-9b03-99d7afba3994_isrForCrosstalkSources_855234_59.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f9b2da64-974e-490e-9b03-99d7afba3994_isrForCrosstalkSources_855234_59.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3fc48a77a24331c343e77b503983d29 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5eb1823a-abf8-42df-857e-c41507920c70_isrForCrosstalkSources_855236_11.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5eb1823a-abf8-42df-857e-c41507920c70_isrForCrosstalkSources_855236_11.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4eaf448498a7da5e90b807c30f71f20 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3f8fb23b-8fa6-4b95-aa0c-120fda607d85_isrForCrosstalkSources_855156_32.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/3f8fb23b-8fa6-4b95-aa0c-120fda607d85_isrForCrosstalkSources_855156_32.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a0cb3c5c18a01cfc70380b48e445341 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fa75006e-d635-4709-82b9-e833770e19b8_isrForCrosstalkSources_855235_45.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fa75006e-d635-4709-82b9-e833770e19b8_isrForCrosstalkSources_855235_45.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00e0490c4c20aa7db5f64c5eee95e82e with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/94218fd2-7715-491e-85b2-0c871cca2ac3_isrForCrosstalkSources_855242_46.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/94218fd2-7715-491e-85b2-0c871cca2ac3_isrForCrosstalkSources_855242_46.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22026ff049e1aa045fdec112304fd019 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a552ecd-32e9-4f85-be05-00ca5077475a_isrForCrosstalkSources_855156_8.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4a552ecd-32e9-4f85-be05-00ca5077475a_isrForCrosstalkSources_855156_8.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5af65a707bf71c86a7d01fe80a97222c with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/1f8728e6-6c21-48e4-96af-fd07b32f339c_isrForCrosstalkSources_855156_6.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/1f8728e6-6c21-48e4-96af-fd07b32f339c_isrForCrosstalkSources_855156_6.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f83b96e38f56842c66bbd5ed13bb2137 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f01ec94f-5116-4a54-b272-4c0eb1e1a7e9_isrForCrosstalkSources_855156_50.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f01ec94f-5116-4a54-b272-4c0eb1e1a7e9_isrForCrosstalkSources_855156_50.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1eef1c46938c20ad8dac5a1d0543e5c with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b949b79-d557-4e9a-8f9e-3ed7122ef5e2_isrForCrosstalkSources_855241_44.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b949b79-d557-4e9a-8f9e-3ed7122ef5e2_isrForCrosstalkSources_855241_44.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9a2a1d86ec5630bdf066402a69bec48 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8c830a3c-97c7-46e8-abeb-2d76748ca361_isrForCrosstalkSources_855236_28.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8c830a3c-97c7-46e8-abeb-2d76748ca361_isrForCrosstalkSources_855236_28.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e65d1fa98d42ca89b865951e9fa1bc with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f8641062-b30e-4cdf-a07d-029e46f62389_isrForCrosstalkSources_855239_62.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f8641062-b30e-4cdf-a07d-029e46f62389_isrForCrosstalkSources_855239_62.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966587334c5d6adfa08d4da736b10141 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ef6d5ad8-08ed-4ef1-b2c7-2e47a762a2ff_isrForCrosstalkSources_855239_29.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ef6d5ad8-08ed-4ef1-b2c7-2e47a762a2ff_isrForCrosstalkSources_855239_29.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4370716f5aae8beaf72a422ec2423bd3 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b8f5b2b9-3f32-4747-bc10-c1618002e85e_isrForCrosstalkSources_855232_27.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/b8f5b2b9-3f32-4747-bc10-c1618002e85e_isrForCrosstalkSources_855232_27.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dce3665f715af7279b2abc773d5542c with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7a3db405-09f7-4c64-a88a-d3b05dff977f_isrForCrosstalkSources_855155_39.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7a3db405-09f7-4c64-a88a-d3b05dff977f_isrForCrosstalkSources_855155_39.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642a7d29b687fc6e427847aa3d345a51 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/49c2d75b-9c8d-4c02-bcce-a8bfc42afa08_isrForCrosstalkSources_855239_20.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/49c2d75b-9c8d-4c02-bcce-a8bfc42afa08_isrForCrosstalkSources_855239_20.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a6c7a767cf67ce94a25f97ebd79ed10 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14e7f5ca-eeca-45d2-b3f9-9c11af3df53b_isrForCrosstalkSources_855240_23.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14e7f5ca-eeca-45d2-b3f9-9c11af3df53b_isrForCrosstalkSources_855240_23.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d3469deeb680ddd912a7ff6747db042 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a0000f44-c1cf-4c73-bb3a-d22fbfde680a_isrForCrosstalkSources_855231_19.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a0000f44-c1cf-4c73-bb3a-d22fbfde680a_isrForCrosstalkSources_855231_19.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40526fd9d62832d66c3a93a3a7d15c69 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7eb89f6b-36d9-4f03-a583-72d9c79bc38a_isrForCrosstalkSources_855237_21.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7eb89f6b-36d9-4f03-a583-72d9c79bc38a_isrForCrosstalkSources_855237_21.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80f7d2610a11b8d11312def965dc9360 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/875b242f-f0a6-491c-b7b8-9f2d632a7933_isrForCrosstalkSources_855156_20.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/875b242f-f0a6-491c-b7b8-9f2d632a7933_isrForCrosstalkSources_855156_20.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87bbfe2dec8f496a76e724a218c53769 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b04f56e4-2a13-4e4e-b035-d1c0b43b7f9a_isrForCrosstalkSources_855154_54.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b04f56e4-2a13-4e4e-b035-d1c0b43b7f9a_isrForCrosstalkSources_855154_54.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdadb7e0bd2334dc0e703ed77746af7e with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/55d59b14-cf0d-4b32-abbb-88fe8306fa24_isrForCrosstalkSources_855154_1.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/55d59b14-cf0d-4b32-abbb-88fe8306fa24_isrForCrosstalkSources_855154_1.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0db4bbb862244485de0acf42c513970e with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/919a4729-d589-4adc-8563-86d8327aacbf_isrForCrosstalkSources_855236_36.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/919a4729-d589-4adc-8563-86d8327aacbf_isrForCrosstalkSources_855236_36.stderr 2024-03-08 10:02:19 proc_lsst.multi:170 [INFO] [multi] job_id 16914768 proc_lsst.multi INFO: [multi] job_id 16914768 2024-03-08 10:02:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-08 10:02:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914768 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914768 parsl.executors.status_handling DEBUG: Launched block 147 on executor multi with job ID 16914768 parsl.executors.status_handling INFO: Allocated block ID 148 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-148 2024-03-08 10:02:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-148 2024-03-08 10:02:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 91 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 91 2024-03-08 10:02:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=148 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dd05e83b16ee8ec156bb4064e5b99f3 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a87aa01b-3643-4bb7-aafb-42f34825ade5_isrForCrosstalkSources_855242_59.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a87aa01b-3643-4bb7-aafb-42f34825ade5_isrForCrosstalkSources_855242_59.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d142d6ebd1e79aca6526c909d64fbc8c with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/11ddc760-2714-45f7-9e1f-e0314e373a75_isrForCrosstalkSources_855235_35.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/11ddc760-2714-45f7-9e1f-e0314e373a75_isrForCrosstalkSources_855235_35.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c89a9aec7081cd713ed6cd4e1d00f9e4 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/9e23c7be-f606-4d0c-abb7-c8cda0a45b40_isrForCrosstalkSources_855241_7.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/9e23c7be-f606-4d0c-abb7-c8cda0a45b40_isrForCrosstalkSources_855241_7.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63fed7803875dcbcdccd4bbfdf2dab16 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b715c15e-3707-4373-b9c3-2ea7efa927fb_isrForCrosstalkSources_855241_12.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b715c15e-3707-4373-b9c3-2ea7efa927fb_isrForCrosstalkSources_855241_12.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fd57a494b1a38a2e07bc31be3315b1d with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/410bbf78-3374-4da4-9444-c613f7e8be8c_isrForCrosstalkSources_855236_42.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/410bbf78-3374-4da4-9444-c613f7e8be8c_isrForCrosstalkSources_855236_42.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ea89eff61d6191566bd4c74ddd04d2 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d26345f-0ae7-461d-82fc-89bacc7712c9_isrForCrosstalkSources_855154_26.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2d26345f-0ae7-461d-82fc-89bacc7712c9_isrForCrosstalkSources_855154_26.stderr 2024-03-08 10:02:20 proc_lsst.multi:170 [INFO] [multi] job_id 16914769 proc_lsst.multi INFO: [multi] job_id 16914769 2024-03-08 10:02:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-08 10:02:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914769 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914769 parsl.executors.status_handling DEBUG: Launched block 148 on executor multi with job ID 16914769 parsl.executors.status_handling INFO: Allocated block ID 149 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-149 2024-03-08 10:02:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-149 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 92 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 92 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=149 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575778281b6c529b6a61103dbf5171f5 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/a23a0a8d-a6ce-4d10-981d-e5e6e69d4305_isrForCrosstalkSources_855154_13.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/a23a0a8d-a6ce-4d10-981d-e5e6e69d4305_isrForCrosstalkSources_855154_13.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74378fdc4476500ffc280f11914217d with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f7b24122-194f-4c62-953f-28068027c044_isrForCrosstalkSources_855234_55.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f7b24122-194f-4c62-953f-28068027c044_isrForCrosstalkSources_855234_55.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 746b70c2c0a3c8aab47d13af651f6852 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/eb142208-4265-4c58-8454-67281af9c8bb_isrForCrosstalkSources_855235_3.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/eb142208-4265-4c58-8454-67281af9c8bb_isrForCrosstalkSources_855235_3.stderr 2024-03-08 10:02:20 proc_lsst.multi:170 [INFO] [multi] job_id 16914770 proc_lsst.multi INFO: [multi] job_id 16914770 2024-03-08 10:02:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-08 10:02:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914770 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914770 parsl.executors.status_handling DEBUG: Launched block 149 on executor multi with job ID 16914770 parsl.executors.status_handling INFO: Allocated block ID 150 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-150 2024-03-08 10:02:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-150 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 93 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 93 2024-03-08 10:02:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=150 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6504074260fa90114591b8c71657727 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c93fc277-50f0-400b-a4ff-d3f1c8607b26_isrForCrosstalkSources_855237_7.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c93fc277-50f0-400b-a4ff-d3f1c8607b26_isrForCrosstalkSources_855237_7.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 968e56c73b58cd5388239752495587ad with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/168437e9-b751-4cc3-b4fb-190b4ac6f579_isrForCrosstalkSources_855154_33.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/168437e9-b751-4cc3-b4fb-190b4ac6f579_isrForCrosstalkSources_855154_33.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ffa3d37176ef6e3d94dc69988520104 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/383b4d4c-9a92-496c-bdd6-58b32de906c6_isrForCrosstalkSources_855231_38.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/383b4d4c-9a92-496c-bdd6-58b32de906c6_isrForCrosstalkSources_855231_38.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0309c7bb8849503582053743aacd7de1 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/798db830-1173-4934-97aa-e06925677a02_isrForCrosstalkSources_855232_2.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/798db830-1173-4934-97aa-e06925677a02_isrForCrosstalkSources_855232_2.stderr 2024-03-08 10:02:21 proc_lsst.multi:170 [INFO] [multi] job_id 16914771 proc_lsst.multi INFO: [multi] job_id 16914771 2024-03-08 10:02:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-08 10:02:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914771 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914771 parsl.executors.status_handling DEBUG: Launched block 150 on executor multi with job ID 16914771 parsl.executors.status_handling INFO: Allocated block ID 151 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-151 2024-03-08 10:02:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-151 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 94 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 94 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=151 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 363132f882ee8109852c3785851a5c14 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c914cda3-b866-4238-ac48-6b9b6ac5b043_isrForCrosstalkSources_855240_53.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c914cda3-b866-4238-ac48-6b9b6ac5b043_isrForCrosstalkSources_855240_53.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1691664e68f3b53e34a4a1a7eb308ab with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/5a848edf-d76a-41a7-ab04-fb004a29be64_isrForCrosstalkSources_855239_11.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/5a848edf-d76a-41a7-ab04-fb004a29be64_isrForCrosstalkSources_855239_11.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03ad803b9602203fefea17f33ca9e009 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a26e7107-edec-4056-81b7-1d5258990696_isrForCrosstalkSources_855235_1.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a26e7107-edec-4056-81b7-1d5258990696_isrForCrosstalkSources_855235_1.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3ecffad28c4a2135f6d59aa02aceccf with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d4e85563-77cb-4c44-8a3f-1c9356ea5717_isrForCrosstalkSources_855235_56.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d4e85563-77cb-4c44-8a3f-1c9356ea5717_isrForCrosstalkSources_855235_56.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ccec982af078bd5ea9d0e653ae03be2 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/bbab30dc-5a2f-44bb-bda6-5b779882bb67_isrForCrosstalkSources_855242_55.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/bbab30dc-5a2f-44bb-bda6-5b779882bb67_isrForCrosstalkSources_855242_55.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c5a0ed3922da97933410fa6ba776d09 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/98f3faea-a9ce-4c61-85e7-540d388e9e49_isrForCrosstalkSources_855242_62.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/98f3faea-a9ce-4c61-85e7-540d388e9e49_isrForCrosstalkSources_855242_62.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15d3be00087820ef9d4b5c28e641e7b4 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/1c538154-72c3-4e15-8701-49f77d3100ca_isrForCrosstalkSources_855231_47.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/1c538154-72c3-4e15-8701-49f77d3100ca_isrForCrosstalkSources_855231_47.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f02c7a1f12a74e55bfec254d72e53f7d with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7d7e7f10-c107-4c05-bbbf-1eb58622a934_isrForCrosstalkSources_855231_48.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/7d7e7f10-c107-4c05-bbbf-1eb58622a934_isrForCrosstalkSources_855231_48.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94010715d86d4c97ace3c6875a909a9e with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6536e09-2b95-461e-af91-a298bf623846_isrForCrosstalkSources_855240_24.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6536e09-2b95-461e-af91-a298bf623846_isrForCrosstalkSources_855240_24.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73674f1c87221cf818e9bec0d14af619 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ce136a19-970e-4281-84a2-c1713432452b_isrForCrosstalkSources_855237_28.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ce136a19-970e-4281-84a2-c1713432452b_isrForCrosstalkSources_855237_28.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55b0b22a8dae845440572f410343e835 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a94c35e3-9a2d-46c2-8bda-e89740db5329_isrForCrosstalkSources_855155_12.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a94c35e3-9a2d-46c2-8bda-e89740db5329_isrForCrosstalkSources_855155_12.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a9ab365a8ef27fa3d7523b2342d830 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/f7d491bf-0339-4014-9a17-49c7567ae08f_isrForCrosstalkSources_855231_1.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/f7d491bf-0339-4014-9a17-49c7567ae08f_isrForCrosstalkSources_855231_1.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55b4fc07f7f426ba406b909223e42977 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/3958c07f-c0ca-4607-8434-5b564a9e3921_isrForCrosstalkSources_855239_37.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/3958c07f-c0ca-4607-8434-5b564a9e3921_isrForCrosstalkSources_855239_37.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97082a1295823c805e25f5b386dad097 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/adea0a8b-7bf1-41a2-8c3c-a8bd60aeac20_isrForCrosstalkSources_855231_15.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/adea0a8b-7bf1-41a2-8c3c-a8bd60aeac20_isrForCrosstalkSources_855231_15.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67d2f1a81e324d4fe636a59ed453e1c9 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/db71e67c-f1b0-402a-8684-931665705fd6_isrForCrosstalkSources_855236_55.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/db71e67c-f1b0-402a-8684-931665705fd6_isrForCrosstalkSources_855236_55.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc6e13fc9c5f73b08e8b2e3eb2c144f6 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a07f1334-11b4-4db6-9ac8-c64c6903eb89_isrForCrosstalkSources_855240_40.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a07f1334-11b4-4db6-9ac8-c64c6903eb89_isrForCrosstalkSources_855240_40.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50ca9f941630bb01bbac42a9655ae4d9 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/790d0944-db5e-430a-84c4-867787e9b911_isrForCrosstalkSources_855238_5.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/790d0944-db5e-430a-84c4-867787e9b911_isrForCrosstalkSources_855238_5.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 096c4c4cbfd75f44f7918ec8628efa2c with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c45e14ad-0d91-4f31-a8d0-618a19d40fcb_isrForCrosstalkSources_855237_10.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c45e14ad-0d91-4f31-a8d0-618a19d40fcb_isrForCrosstalkSources_855237_10.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02a2c45447585b12d11f4d6c824c021a with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/527efd63-f35a-48ee-bb27-05289e1d7a0b_isrForCrosstalkSources_855231_24.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/527efd63-f35a-48ee-bb27-05289e1d7a0b_isrForCrosstalkSources_855231_24.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08502c22e14fafa8c18c3fa9aca1edcd with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/444a57c6-2981-4098-9417-7a3d19e84cba_isrForCrosstalkSources_855242_26.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/444a57c6-2981-4098-9417-7a3d19e84cba_isrForCrosstalkSources_855242_26.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26cd605f327c34eea712003dd8427e4 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f825c3c4-7b9e-44de-beba-b0ff9a61ca49_isrForCrosstalkSources_855239_47.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f825c3c4-7b9e-44de-beba-b0ff9a61ca49_isrForCrosstalkSources_855239_47.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bdf53e3d5bfb2967007a1942a43aeda with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a2c11462-b1ef-4883-87e1-6125e2a07d14_isrForCrosstalkSources_855236_33.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/a2c11462-b1ef-4883-87e1-6125e2a07d14_isrForCrosstalkSources_855236_33.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e55f91bc91c116794448f18cea1425a5 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e9e42015-6c85-4c5a-aff6-e5381918364b_isrForCrosstalkSources_855154_29.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e9e42015-6c85-4c5a-aff6-e5381918364b_isrForCrosstalkSources_855154_29.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a0cf5dd7ffdbc2637b8202c3f94e7ec with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c11652e0-a895-4f64-81da-53f203406628_isrForCrosstalkSources_855231_26.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/c11652e0-a895-4f64-81da-53f203406628_isrForCrosstalkSources_855231_26.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73a7294ac4d5042577472f9f73683ba6 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd84a1c4-d2d8-473a-b6a9-4ca70aac576a_isrForCrosstalkSources_855154_2.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dd84a1c4-d2d8-473a-b6a9-4ca70aac576a_isrForCrosstalkSources_855154_2.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 877056c6ae000c0c81c895d76c417a82 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/79acced6-6c82-42b1-baa2-abba42fe8bd5_isrForCrosstalkSources_855232_46.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/79acced6-6c82-42b1-baa2-abba42fe8bd5_isrForCrosstalkSources_855232_46.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b818cdd7ae930041448bdfd46d9f28 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/0905e600-aee7-4216-a01a-e1cfefebd0c5_isrForCrosstalkSources_855240_14.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/0905e600-aee7-4216-a01a-e1cfefebd0c5_isrForCrosstalkSources_855240_14.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11c53ad79047ca497fa3f868cce8859a with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2050d7b8-740d-4cbe-83a7-1d38ef8b1389_isrForCrosstalkSources_855154_46.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2050d7b8-740d-4cbe-83a7-1d38ef8b1389_isrForCrosstalkSources_855154_46.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb72f6cb9ef010b06f29a626507217c9 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4fc4314d-fcf5-4294-aeaf-8b6d32359cd0_isrForCrosstalkSources_855242_13.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4fc4314d-fcf5-4294-aeaf-8b6d32359cd0_isrForCrosstalkSources_855242_13.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 669fb647c2c09c03c56015e5dd937ffc with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6bd0f52e-96ad-494b-940a-365c8b45071a_isrForCrosstalkSources_855156_2.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/6bd0f52e-96ad-494b-940a-365c8b45071a_isrForCrosstalkSources_855156_2.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24308cb8ce555197671fe7faaf6136b with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6f730530-4331-4c65-9467-7ea06264cef0_isrForCrosstalkSources_855155_48.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6f730530-4331-4c65-9467-7ea06264cef0_isrForCrosstalkSources_855155_48.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af95b01b02faa397c85cfdf8151b7c8a with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/88c6a692-b32c-451b-bf69-a1683d153131_isrForCrosstalkSources_855240_62.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/88c6a692-b32c-451b-bf69-a1683d153131_isrForCrosstalkSources_855240_62.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73bac566590cf2d12aaf50284f11270f with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/34b908df-9b82-4b92-b9f5-1c94ebf59ef3_isrForCrosstalkSources_855234_14.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/34b908df-9b82-4b92-b9f5-1c94ebf59ef3_isrForCrosstalkSources_855234_14.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f762b365dcf089f343f2b1f8c51610dc with result from task 413 2024-03-08 10:02:21 proc_lsst.multi:170 [INFO] [multi] job_id 16914772 proc_lsst.multi INFO: [multi] job_id 16914772 2024-03-08 10:02:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-08 10:02:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914772 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914772 parsl.executors.status_handling DEBUG: Launched block 151 on executor multi with job ID 16914772 parsl.executors.status_handling INFO: Allocated block ID 152 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-152 2024-03-08 10:02:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-152 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 95 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 95 2024-03-08 10:02:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=152 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/419118bf-abb1-4063-8542-b16acd4b9850_isrForCrosstalkSources_855232_55.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/419118bf-abb1-4063-8542-b16acd4b9850_isrForCrosstalkSources_855232_55.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70b91f2d44789480746dc7e5fb8003a9 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/22625fd5-104d-4d28-859e-032808520819_isrForCrosstalkSources_855238_27.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/22625fd5-104d-4d28-859e-032808520819_isrForCrosstalkSources_855238_27.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f55da4768ebd8a0a9daaf6131a11aa with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7b203558-f083-40d9-934c-84badd48ecd6_isrForCrosstalkSources_855236_19.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7b203558-f083-40d9-934c-84badd48ecd6_isrForCrosstalkSources_855236_19.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 150316ff2846ba8937349c73b57f3a66 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d54844df-f5a7-4052-b551-e045efbf968d_isrForCrosstalkSources_855233_59.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d54844df-f5a7-4052-b551-e045efbf968d_isrForCrosstalkSources_855233_59.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d24e776f33416a4cbe53fd5f6b2bce1 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b474a55d-1ab2-48fd-825e-5d7b9f242395_isrForCrosstalkSources_855235_24.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b474a55d-1ab2-48fd-825e-5d7b9f242395_isrForCrosstalkSources_855235_24.stderr 2024-03-08 10:02:22 proc_lsst.multi:170 [INFO] [multi] job_id 16914773 proc_lsst.multi INFO: [multi] job_id 16914773 2024-03-08 10:02:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-08 10:02:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914773 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914773 parsl.executors.status_handling DEBUG: Launched block 152 on executor multi with job ID 16914773 parsl.executors.status_handling INFO: Allocated block ID 153 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-153 2024-03-08 10:02:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-153 2024-03-08 10:02:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 96 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 96 2024-03-08 10:02:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=153 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c675acde94f1f78e5b4dd72d7274df1 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/099d780e-6573-474a-8d14-7e6a6b88e980_isrForCrosstalkSources_855233_35.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/099d780e-6573-474a-8d14-7e6a6b88e980_isrForCrosstalkSources_855233_35.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43ac83fe8abb091219d7f2dfb3d30b21 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/43704e7b-f22d-4706-b7ee-08bcd42a514e_isrForCrosstalkSources_855235_10.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/43704e7b-f22d-4706-b7ee-08bcd42a514e_isrForCrosstalkSources_855235_10.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ca9df6fb3b0407b52cfb669fd3e9a5 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/58a64cad-cc71-4ad5-afc0-4272707e729b_isrForCrosstalkSources_855241_23.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/58a64cad-cc71-4ad5-afc0-4272707e729b_isrForCrosstalkSources_855241_23.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63debddf92497e30883b8c0b3215a3fd with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/646ae36a-1052-4a0a-92b1-e60f74c129b0_isrForCrosstalkSources_855234_22.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/646ae36a-1052-4a0a-92b1-e60f74c129b0_isrForCrosstalkSources_855234_22.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6bc078ebae12a8410099a4e31f4c430 with result from task 242 2024-03-08 10:02:23 proc_lsst.multi:170 [INFO] [multi] job_id 16914774 proc_lsst.multi INFO: [multi] job_id 16914774 2024-03-08 10:02:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-08 10:02:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914774 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914774 parsl.executors.status_handling DEBUG: Launched block 153 on executor multi with job ID 16914774 parsl.executors.status_handling INFO: Allocated block ID 154 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-154 2024-03-08 10:02:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-154 2024-03-08 10:02:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 97 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 97 2024-03-08 10:02:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=154 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a816879e-4e24-431c-ad61-d76c76e34910_isrForCrosstalkSources_855240_16.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/a816879e-4e24-431c-ad61-d76c76e34910_isrForCrosstalkSources_855240_16.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5af40706c84ba07ccd5ad76ea768919a with result from task 238 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/0d766dc9-a6d5-451d-ab5c-0e4f5ad56121_isrForCrosstalkSources_855236_10.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/0d766dc9-a6d5-451d-ab5c-0e4f5ad56121_isrForCrosstalkSources_855236_10.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc76388d9063ca1eb280028e706f0cf0 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cee0d7d2-3981-45d3-96d2-45d1820bab50_isrForCrosstalkSources_855241_10.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/cee0d7d2-3981-45d3-96d2-45d1820bab50_isrForCrosstalkSources_855241_10.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 547c6646c062a7de719b9cdd65be63f0 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/bd83980e-71e4-472c-b510-a9302ac8487b_isrForCrosstalkSources_855234_17.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/bd83980e-71e4-472c-b510-a9302ac8487b_isrForCrosstalkSources_855234_17.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2105aeb5d5a98475686228fbe28d7983 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/79f87437-5365-45e5-9c07-1cef594f42d2_isrForCrosstalkSources_855156_35.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/79f87437-5365-45e5-9c07-1cef594f42d2_isrForCrosstalkSources_855156_35.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c7bb515103ff38a35c8ada5eb80f9db with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/26bacb6c-ad5f-4e52-a21e-1d0ac8b91fe3_isrForCrosstalkSources_855236_35.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/26bacb6c-ad5f-4e52-a21e-1d0ac8b91fe3_isrForCrosstalkSources_855236_35.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f3bbeb12cee53e8f99d9434e24d4225 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/42d77067-b130-4faf-bf77-1cc39db43c68_isrForCrosstalkSources_855234_58.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/42d77067-b130-4faf-bf77-1cc39db43c68_isrForCrosstalkSources_855234_58.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00bdc0c6c05511711abce53af846e654 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/50f44755-afa3-42c6-bc1f-e7e99ef2f3b2_isrForCrosstalkSources_855154_12.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/50f44755-afa3-42c6-bc1f-e7e99ef2f3b2_isrForCrosstalkSources_855154_12.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e65362246129ef05859c174337577fdb with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e5383b5a-201b-4b77-bab6-e62ddf825e08_isrForCrosstalkSources_855242_6.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e5383b5a-201b-4b77-bab6-e62ddf825e08_isrForCrosstalkSources_855242_6.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 461c827e0e4094602b161fd5d57f9416 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc3aac43-8214-4b0b-b06e-fb98499e096d_isrForCrosstalkSources_855235_27.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fc3aac43-8214-4b0b-b06e-fb98499e096d_isrForCrosstalkSources_855235_27.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83bdac19513c1f2e14e2572dd64c1405 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8a404fd5-849e-4715-ab75-d5c93c99f1e9_isrForCrosstalkSources_855234_36.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8a404fd5-849e-4715-ab75-d5c93c99f1e9_isrForCrosstalkSources_855234_36.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d347d25170db8ec352d0de1b4fb67c with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/69efa9fa-6565-4ce2-b4b5-ed980c584b78_isrForCrosstalkSources_855236_39.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/69efa9fa-6565-4ce2-b4b5-ed980c584b78_isrForCrosstalkSources_855236_39.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4566ae375caeaf65cf8a62309e377b6 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/86417a52-f1e2-45e9-8885-467e9cc9b34c_isrForCrosstalkSources_855237_45.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/86417a52-f1e2-45e9-8885-467e9cc9b34c_isrForCrosstalkSources_855237_45.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc80a1aa88bd983780c5f615cafaea65 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/19dac0f7-28c5-4d1f-a09d-e20a7bc651b3_isrForCrosstalkSources_855236_54.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/19dac0f7-28c5-4d1f-a09d-e20a7bc651b3_isrForCrosstalkSources_855236_54.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6374b4151d853dc279db3a39e4cbec0f with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/fb2e9e81-1bd6-4879-94c2-5df987f2bea6_isrForCrosstalkSources_855232_51.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/fb2e9e81-1bd6-4879-94c2-5df987f2bea6_isrForCrosstalkSources_855232_51.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d848dcd4a953c4008f7692b66191176 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/09dc70b1-7250-4563-a782-e9d095d66f95_isrForCrosstalkSources_855154_4.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/09dc70b1-7250-4563-a782-e9d095d66f95_isrForCrosstalkSources_855154_4.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e18137a0803867780b86f3b082f9337 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9de9a0d6-5ae9-48a9-b7fd-042687fdbc86_isrForCrosstalkSources_855154_15.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9de9a0d6-5ae9-48a9-b7fd-042687fdbc86_isrForCrosstalkSources_855154_15.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb69cabc4f22bb0747deb844fbe8c7bc with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d18f768e-efb2-4552-bef8-bc944f45572a_isrForCrosstalkSources_855240_59.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d18f768e-efb2-4552-bef8-bc944f45572a_isrForCrosstalkSources_855240_59.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7f1a32a3236f9d66793280064e8eb49 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/17978ef6-5e21-48ef-ba48-c934f7c012f2_isrForCrosstalkSources_855242_31.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/17978ef6-5e21-48ef-ba48-c934f7c012f2_isrForCrosstalkSources_855242_31.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c71173a9cd6c7054f25bc69fbe0d35a3 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ab01297-239d-4a85-ba31-567d4ac64649_isrForCrosstalkSources_855234_21.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2ab01297-239d-4a85-ba31-567d4ac64649_isrForCrosstalkSources_855234_21.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1b18ff71ff82fced2e91b7fc6af5403 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/25c25ab0-4877-4a1e-9eb3-4a8bce3b88f3_isrForCrosstalkSources_855234_3.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/25c25ab0-4877-4a1e-9eb3-4a8bce3b88f3_isrForCrosstalkSources_855234_3.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15f10e27fc82fcf8a3444b0a088c0f8 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d9e87ab0-c8df-4c90-8a85-5c7044c6b538_isrForCrosstalkSources_855154_47.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d9e87ab0-c8df-4c90-8a85-5c7044c6b538_isrForCrosstalkSources_855154_47.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ad8e76b603aba9b0a7211d4e5b3ec84 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/19277ea6-6202-4c31-a507-98b472f58865_isrForCrosstalkSources_855235_49.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/19277ea6-6202-4c31-a507-98b472f58865_isrForCrosstalkSources_855235_49.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 052127d772da8612d873214431c9629e with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3531398-a054-489a-8856-c977138875ab_isrForCrosstalkSources_855235_17.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3531398-a054-489a-8856-c977138875ab_isrForCrosstalkSources_855235_17.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17132ba6233463b7c11d02a8ceed8634 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f19614c0-4405-4a62-b103-52ad4a14af6e_isrForCrosstalkSources_855242_4.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f19614c0-4405-4a62-b103-52ad4a14af6e_isrForCrosstalkSources_855242_4.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2529091968c3bbf6b926b9744472d3bb with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d4c6a4db-b391-4dcf-936c-48ac8f29bb53_isrForCrosstalkSources_855232_20.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d4c6a4db-b391-4dcf-936c-48ac8f29bb53_isrForCrosstalkSources_855232_20.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fed5912cd2db33f31ca95a2cedee5973 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/61150b6f-73a6-47ad-bcdf-27cee42e8478_isrForCrosstalkSources_855236_56.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/61150b6f-73a6-47ad-bcdf-27cee42e8478_isrForCrosstalkSources_855236_56.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b728e87f6a9142f6bab6eb497c66e5d with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/51cbb6fd-3b64-4f71-ad86-15738e87ce5f_isrForCrosstalkSources_855236_38.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/51cbb6fd-3b64-4f71-ad86-15738e87ce5f_isrForCrosstalkSources_855236_38.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd586c948805168f71dc77337e101161 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/29e3e857-4783-46e9-b97a-a6bc0adacaaf_isrForCrosstalkSources_855241_45.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/29e3e857-4783-46e9-b97a-a6bc0adacaaf_isrForCrosstalkSources_855241_45.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8253a82b276d006864189d981ff2b904 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c884bb95-5939-45ec-ab31-6bcb63a9b26e_isrForCrosstalkSources_855154_25.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c884bb95-5939-45ec-ab31-6bcb63a9b26e_isrForCrosstalkSources_855154_25.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f2fcdab54bcdb14196eceafe79ec05 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d2c834ff-a195-4e96-91db-1fd556639d20_isrForCrosstalkSources_855231_21.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d2c834ff-a195-4e96-91db-1fd556639d20_isrForCrosstalkSources_855231_21.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b7439e558ae5cf382acceea5c71f201 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9f0c673f-76c2-434e-ba17-ba987e2be7e1_isrForCrosstalkSources_855232_15.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/9f0c673f-76c2-434e-ba17-ba987e2be7e1_isrForCrosstalkSources_855232_15.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f04d24aa319f40b02897e4f7e2609e1 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2de19804-fb88-476e-bb82-a1ad67804f53_isrForCrosstalkSources_855239_13.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2de19804-fb88-476e-bb82-a1ad67804f53_isrForCrosstalkSources_855239_13.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f06167dd53228374da07fa31fd57334 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/41602420-cdb7-40e6-9ce9-2bf06fc2caae_isrForCrosstalkSources_855155_2.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/41602420-cdb7-40e6-9ce9-2bf06fc2caae_isrForCrosstalkSources_855155_2.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a65b471500a8751a39d207f7ca2c8e with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8e53925-1a4b-4436-9d4e-3b5265c3402d_isrForCrosstalkSources_855155_10.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b8e53925-1a4b-4436-9d4e-3b5265c3402d_isrForCrosstalkSources_855155_10.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b06e060c1a9efa62e064b54e45cfcabb with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e0a2eb93-503c-46f1-8cfc-4d0ada6daa58_isrForCrosstalkSources_855241_6.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e0a2eb93-503c-46f1-8cfc-4d0ada6daa58_isrForCrosstalkSources_855241_6.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d26125b08a5473c767da15cfd818547c with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/596cd7a3-e5dc-4a6a-8435-0cfeba205f1d_isrForCrosstalkSources_855240_25.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/596cd7a3-e5dc-4a6a-8435-0cfeba205f1d_isrForCrosstalkSources_855240_25.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41f4e0365b95045338e40e8dd6f8c26d with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ef62c870-5539-420b-83c0-901649aeea0d_isrForCrosstalkSources_855233_17.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ef62c870-5539-420b-83c0-901649aeea0d_isrForCrosstalkSources_855233_17.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd7363b1c92e08d0bb178da82132dbdc with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0f102005-8229-4a85-be2a-c94099fbc670_isrForCrosstalkSources_855234_20.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0f102005-8229-4a85-be2a-c94099fbc670_isrForCrosstalkSources_855234_20.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d86a4d5f6b6cc068b1b26e14ab80cb1f with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a94dbcd0-a4a3-4f65-a9f5-d571693caf51_isrForCrosstalkSources_855231_28.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/a94dbcd0-a4a3-4f65-a9f5-d571693caf51_isrForCrosstalkSources_855231_28.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9cdb658b14cad275c3b6d2d00415e87 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/27ff2633-cd87-4e6b-8878-32c66e400f3d_isrForCrosstalkSources_855236_30.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/27ff2633-cd87-4e6b-8878-32c66e400f3d_isrForCrosstalkSources_855236_30.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c4758cfc985c2db99f263cd7b3b5db4 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a5d9a0f-9b0d-4fb4-9e10-2767eb1624d5_isrForCrosstalkSources_855237_52.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a5d9a0f-9b0d-4fb4-9e10-2767eb1624d5_isrForCrosstalkSources_855237_52.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71166ebbac9d0e48adc86719781e1813 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9a852e51-7613-4c14-80df-5d96863521d1_isrForCrosstalkSources_855240_15.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/9a852e51-7613-4c14-80df-5d96863521d1_isrForCrosstalkSources_855240_15.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f8462e43d16e4e97003d1d9e81bb21f with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f8f956fe-d5c9-4c5a-b3b7-908214249f93_isrForCrosstalkSources_855155_3.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/f8f956fe-d5c9-4c5a-b3b7-908214249f93_isrForCrosstalkSources_855155_3.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7771fcb142c88b4c8241bf4d150a9378 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3b0e667d-5cc1-407a-bc64-f4b712f1ceac_isrForCrosstalkSources_855234_48.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3b0e667d-5cc1-407a-bc64-f4b712f1ceac_isrForCrosstalkSources_855234_48.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97467e854461a4d5018ca93b9b81ace5 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8e8b2326-323b-46da-86b8-edf2e6a61b95_isrForCrosstalkSources_855154_50.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8e8b2326-323b-46da-86b8-edf2e6a61b95_isrForCrosstalkSources_855154_50.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9225ed30d7ecb647650d53f20ebf5f8d with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/98edcfc3-8180-41ca-91e8-7677426db419_isrForCrosstalkSources_855236_3.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/98edcfc3-8180-41ca-91e8-7677426db419_isrForCrosstalkSources_855236_3.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 132c719817784594c7e13838221756ef with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/cea92e5d-3c49-4f8a-9f2c-e79138ccae01_isrForCrosstalkSources_855235_11.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/cea92e5d-3c49-4f8a-9f2c-e79138ccae01_isrForCrosstalkSources_855235_11.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 406d28becc66542b570b30e345dc0700 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d2753ee3-e266-4c5b-9fd1-d5ec95666787_isrForCrosstalkSources_855238_36.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d2753ee3-e266-4c5b-9fd1-d5ec95666787_isrForCrosstalkSources_855238_36.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f33153922803e28060765abfd8e6c587 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/59079c17-83eb-4022-997a-b9dcd7436943_isrForCrosstalkSources_855239_15.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/59079c17-83eb-4022-997a-b9dcd7436943_isrForCrosstalkSources_855239_15.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da32e6179cfc00655426d543bf3551ab with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c33ac37f-b7c9-42ab-a846-75ebf84bed13_isrForCrosstalkSources_855237_46.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c33ac37f-b7c9-42ab-a846-75ebf84bed13_isrForCrosstalkSources_855237_46.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16be16eedce12398a5e0412bc1b24e59 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8f6c9629-f2c7-4655-8710-6b90ef1317c3_isrForCrosstalkSources_855237_34.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8f6c9629-f2c7-4655-8710-6b90ef1317c3_isrForCrosstalkSources_855237_34.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 814477059d0da3025b79c694dbc42903 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/b3c48d53-e9b6-4fdd-89cc-ece47604a381_isrForCrosstalkSources_855237_19.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/b3c48d53-e9b6-4fdd-89cc-ece47604a381_isrForCrosstalkSources_855237_19.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c1810b783c66225fe70da2065c7783c with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/4cf480c1-6157-47ca-85df-b9d7bc22940d_isrForCrosstalkSources_855232_35.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/4cf480c1-6157-47ca-85df-b9d7bc22940d_isrForCrosstalkSources_855232_35.stderr parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e21c41a90249cf13e38d37ef4000cde9 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/6f78584e-d723-4b55-a558-ff17d5e2690e_isrForCrosstalkSources_855234_10.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/6f78584e-d723-4b55-a558-ff17d5e2690e_isrForCrosstalkSources_855234_10.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e00c411eeb16c2ff7715ecb6b4a7ec5 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/68a3f9eb-8155-494b-8c2c-ba896b654ff3_isrForCrosstalkSources_855156_53.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/68a3f9eb-8155-494b-8c2c-ba896b654ff3_isrForCrosstalkSources_855156_53.stderr parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c718d63dc9d68c6ea39dfedc637ff103 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0c97fd68-fff1-436f-9704-4fd469ecc057_isrForCrosstalkSources_855155_60.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0c97fd68-fff1-436f-9704-4fd469ecc057_isrForCrosstalkSources_855155_60.stderr 2024-03-08 10:02:24 proc_lsst.multi:170 [INFO] [multi] job_id 16914775 proc_lsst.multi INFO: [multi] job_id 16914775 2024-03-08 10:02:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-08 10:02:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914775 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914775 parsl.executors.status_handling DEBUG: Launched block 154 on executor multi with job ID 16914775 parsl.executors.status_handling INFO: Allocated block ID 155 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-155 2024-03-08 10:02:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-155 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 98 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 98 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=155 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:24 proc_lsst.multi:170 [INFO] [multi] job_id 16914776 proc_lsst.multi INFO: [multi] job_id 16914776 2024-03-08 10:02:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-08 10:02:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914776 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914776 parsl.executors.status_handling DEBUG: Launched block 155 on executor multi with job ID 16914776 parsl.executors.status_handling INFO: Allocated block ID 156 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-156 2024-03-08 10:02:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-156 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 99 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 99 2024-03-08 10:02:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=156 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b54c2a23c0426baf42386155548561d with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9ae5ce35-5153-48ca-ade7-679d3e65657a_isrForCrosstalkSources_855233_22.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9ae5ce35-5153-48ca-ade7-679d3e65657a_isrForCrosstalkSources_855233_22.stderr 2024-03-08 10:02:25 proc_lsst.multi:170 [INFO] [multi] job_id 16914777 proc_lsst.multi INFO: [multi] job_id 16914777 2024-03-08 10:02:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-08 10:02:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914777 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914777 parsl.executors.status_handling DEBUG: Launched block 156 on executor multi with job ID 16914777 parsl.executors.status_handling INFO: Allocated block ID 157 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-157 2024-03-08 10:02:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-157 2024-03-08 10:02:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 100 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 100 2024-03-08 10:02:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=157 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95fad40275e03e131f0f862c624f32d with result from task 509 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f26822f5-cf30-463e-8bc0-707c335afa49_isrForCrosstalkSources_855232_62.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f26822f5-cf30-463e-8bc0-707c335afa49_isrForCrosstalkSources_855232_62.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c296491df6d6ccd6cd09e937ae0fbf8 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7ebd90f3-bb64-4dac-8876-dc9c72d9a688_isrForCrosstalkSources_855155_24.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7ebd90f3-bb64-4dac-8876-dc9c72d9a688_isrForCrosstalkSources_855155_24.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10ff623f8fcb714a1bc2dc199e410652 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b285ad8b-e571-4926-8f54-bcb5b238ea27_isrForCrosstalkSources_855239_61.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b285ad8b-e571-4926-8f54-bcb5b238ea27_isrForCrosstalkSources_855239_61.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328ed8db7b71452cf18535109c2d53b6 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ca7bf9d2-e28c-48e5-95b0-2992cac1cd86_isrForCrosstalkSources_855155_6.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ca7bf9d2-e28c-48e5-95b0-2992cac1cd86_isrForCrosstalkSources_855155_6.stderr 2024-03-08 10:02:26 proc_lsst.multi:170 [INFO] [multi] job_id 16914778 proc_lsst.multi INFO: [multi] job_id 16914778 2024-03-08 10:02:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-08 10:02:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914778 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914778 parsl.executors.status_handling DEBUG: Launched block 157 on executor multi with job ID 16914778 parsl.executors.status_handling INFO: Allocated block ID 158 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-158 2024-03-08 10:02:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-158 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 101 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 101 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=158 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6d5abbb09ac231028259e3cb26eb5e4 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/47f54081-8311-4eff-b741-15ac14dc1c41_isrForCrosstalkSources_855154_28.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/47f54081-8311-4eff-b741-15ac14dc1c41_isrForCrosstalkSources_855154_28.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d1b1dd3cc9216a13c838c2f3fc51db3 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/386e3382-b042-4f99-b14c-ee6860587787_isrForCrosstalkSources_855234_32.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/386e3382-b042-4f99-b14c-ee6860587787_isrForCrosstalkSources_855234_32.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f3478bfc7b814286b4b8f4eac5069e7 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2be9391f-4428-4d81-af61-ada8b9b6a855_isrForCrosstalkSources_855242_25.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2be9391f-4428-4d81-af61-ada8b9b6a855_isrForCrosstalkSources_855242_25.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb65c9ac757d75819c84c5ff5f934796 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8f47e77c-ca05-46c6-b573-561b698f8e35_isrForCrosstalkSources_855238_32.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8f47e77c-ca05-46c6-b573-561b698f8e35_isrForCrosstalkSources_855238_32.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a72f262650f57f4d59adb41c7b57043 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20d3f772-e8cc-4670-8896-32169ae1310a_isrForCrosstalkSources_855156_17.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/20d3f772-e8cc-4670-8896-32169ae1310a_isrForCrosstalkSources_855156_17.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 266cffb8eac845741aadcef3b1bc62ab with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/636a06dd-85ca-40f9-864c-2db4822e6cae_isrForCrosstalkSources_855237_17.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/636a06dd-85ca-40f9-864c-2db4822e6cae_isrForCrosstalkSources_855237_17.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c709c07cc7192d4862d0d555da7e4d89 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/03712f63-f489-4365-9589-829dc1883971_isrForCrosstalkSources_855240_46.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/03712f63-f489-4365-9589-829dc1883971_isrForCrosstalkSources_855240_46.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6f14f4f6a2e5adf28f066f21a4a1d2c with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/501f4bd8-dac8-4768-9ecf-feea2247f0b5_isrForCrosstalkSources_855156_30.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/501f4bd8-dac8-4768-9ecf-feea2247f0b5_isrForCrosstalkSources_855156_30.stderr parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20516e50846cb6f4f775f4b893725d49 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/be9f6d54-6b0d-4aca-8cc5-4e215e24ae22_isrForCrosstalkSources_855233_56.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/be9f6d54-6b0d-4aca-8cc5-4e215e24ae22_isrForCrosstalkSources_855233_56.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b22d0b1277150b3f621da32ba41f2e28 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/cad6f813-b4e8-4dfb-bef2-d56133f5a488_isrForCrosstalkSources_855231_7.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/cad6f813-b4e8-4dfb-bef2-d56133f5a488_isrForCrosstalkSources_855231_7.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 138eb2b7ae1308bb68bbc10a4bbc15e5 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ab1a2a57-3105-4ee1-aa23-eb643538cd6a_isrForCrosstalkSources_855155_49.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ab1a2a57-3105-4ee1-aa23-eb643538cd6a_isrForCrosstalkSources_855155_49.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 686721e184f37910d761011a0b7f8ff5 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56e409ea-9eb0-4cbd-9434-c83c958c3673_isrForCrosstalkSources_855237_29.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/56e409ea-9eb0-4cbd-9434-c83c958c3673_isrForCrosstalkSources_855237_29.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04a1cab5bb3adf9cb7ca2643a482b0d2 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/cf9ccfd6-c407-40a2-b494-a2d584e2272d_isrForCrosstalkSources_855234_34.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/cf9ccfd6-c407-40a2-b494-a2d584e2272d_isrForCrosstalkSources_855234_34.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0961ea11f330c5b1bab768e593c755cb with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fe2dac1f-b3c0-4d3d-84a9-27aac4d17a3e_isrForCrosstalkSources_855235_38.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/fe2dac1f-b3c0-4d3d-84a9-27aac4d17a3e_isrForCrosstalkSources_855235_38.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5159ded7da463a52e9f1b526b4f1d4c7 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/aad694cd-adf3-4e91-a912-84e4b07d8fb6_isrForCrosstalkSources_855155_14.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/aad694cd-adf3-4e91-a912-84e4b07d8fb6_isrForCrosstalkSources_855155_14.stderr 2024-03-08 10:02:26 proc_lsst.multi:170 [INFO] [multi] job_id 16914779 proc_lsst.multi INFO: [multi] job_id 16914779 2024-03-08 10:02:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-08 10:02:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914779 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914779 parsl.executors.status_handling DEBUG: Launched block 158 on executor multi with job ID 16914779 parsl.executors.status_handling INFO: Allocated block ID 159 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-159 2024-03-08 10:02:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-159 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 102 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 102 2024-03-08 10:02:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=159 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f440cbefdfdef73bbd059f96b18d1e19 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/501d1766-6943-446d-b2cd-6d7df7251098_isrForCrosstalkSources_855232_9.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/501d1766-6943-446d-b2cd-6d7df7251098_isrForCrosstalkSources_855232_9.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38bb73bd67b31d8cc1acaab8ade4d738 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/416f9eea-7a4c-4bf8-8592-5767ca8a2cd0_isrForCrosstalkSources_855235_42.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/416f9eea-7a4c-4bf8-8592-5767ca8a2cd0_isrForCrosstalkSources_855235_42.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72194c6ad5eb860ad6e9fe1a6a6a6ed8 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a7ced4f2-fd17-4061-b874-858572c66138_isrForCrosstalkSources_855156_31.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a7ced4f2-fd17-4061-b874-858572c66138_isrForCrosstalkSources_855156_31.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9aea70654d3cd207c90cb8a62fdeece5 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/daf21744-aa88-44a5-b63e-8956b8281709_isrForCrosstalkSources_855240_12.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/daf21744-aa88-44a5-b63e-8956b8281709_isrForCrosstalkSources_855240_12.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dbb0490e72f5282652a2b3c97cdd57f with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b20d63d-c126-4ef9-9454-79fa30a74c50_isrForCrosstalkSources_855241_17.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3b20d63d-c126-4ef9-9454-79fa30a74c50_isrForCrosstalkSources_855241_17.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7833da8daadb08b7691ef6161194c63 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/7517041e-51ce-418f-90af-eb8baef0d007_isrForCrosstalkSources_855242_41.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/7517041e-51ce-418f-90af-eb8baef0d007_isrForCrosstalkSources_855242_41.stderr parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d620d60e9d58c88905ab60d1dd6da051 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/efc84dee-948a-4dcc-bb70-057c2a1315bb_isrForCrosstalkSources_855232_25.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/efc84dee-948a-4dcc-bb70-057c2a1315bb_isrForCrosstalkSources_855232_25.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91383ad76dee919fa4aa1cf93991bfa3 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/af133832-1fa2-4a77-bd41-0930d7e890c8_isrForCrosstalkSources_855239_43.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/af133832-1fa2-4a77-bd41-0930d7e890c8_isrForCrosstalkSources_855239_43.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9357ea507dd03a40f2cd003297a111fb with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/403e657a-0f44-4035-b957-8492863a7117_isrForCrosstalkSources_855156_28.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/403e657a-0f44-4035-b957-8492863a7117_isrForCrosstalkSources_855156_28.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691f1267ab716d7d9a4a996a907b0231 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6ed331a-fc0f-4efa-ae6e-6192d7d50ac2_isrForCrosstalkSources_855232_30.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f6ed331a-fc0f-4efa-ae6e-6192d7d50ac2_isrForCrosstalkSources_855232_30.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 796865f5ca71e9363bced37c11b7f4bb with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6d2012c1-5288-47a2-a0d7-dd09979555a3_isrForCrosstalkSources_855154_19.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6d2012c1-5288-47a2-a0d7-dd09979555a3_isrForCrosstalkSources_855154_19.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15cd9b34a9c6de1fa0ba643c22837039 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ccba2a76-8795-4604-8212-229dd94f927a_isrForCrosstalkSources_855242_8.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ccba2a76-8795-4604-8212-229dd94f927a_isrForCrosstalkSources_855242_8.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3321cf3c060d7e5377da9edc2498ed9 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0a7468ed-b63d-4955-afa6-d985a59b3026_isrForCrosstalkSources_855242_9.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0a7468ed-b63d-4955-afa6-d985a59b3026_isrForCrosstalkSources_855242_9.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fdc02e7a50e52460c09f576f859c608 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/daa83ac4-9e20-491d-a7ef-c457f949c9b2_isrForCrosstalkSources_855236_23.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/daa83ac4-9e20-491d-a7ef-c457f949c9b2_isrForCrosstalkSources_855236_23.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ed7c9c3e977fd0d2debb018281bd7ef with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/66373349-6031-481f-ab72-7894f5a109c0_isrForCrosstalkSources_855233_7.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/66373349-6031-481f-ab72-7894f5a109c0_isrForCrosstalkSources_855233_7.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b344e2b33550584d5f1b592c78eeb7eb with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1db546ce-5808-4184-9c12-cfa5b81196ac_isrForCrosstalkSources_855155_30.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1db546ce-5808-4184-9c12-cfa5b81196ac_isrForCrosstalkSources_855155_30.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01756181441ca978b80ef39aeaf4c514 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24c7532d-bf87-457f-a73b-bf7d7f2a7034_isrForCrosstalkSources_855231_13.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/24c7532d-bf87-457f-a73b-bf7d7f2a7034_isrForCrosstalkSources_855231_13.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f55adae8536e46849eedf7a842edabf2 with result from task 491 2024-03-08 10:02:27 proc_lsst.multi:170 [INFO] [multi] job_id 16914780 proc_lsst.multi INFO: [multi] job_id 16914780 2024-03-08 10:02:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-08 10:02:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914780 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914780 parsl.executors.status_handling DEBUG: Launched block 159 on executor multi with job ID 16914780 parsl.executors.status_handling INFO: Allocated block ID 160 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-160 2024-03-08 10:02:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-160 2024-03-08 10:02:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 103 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 103 2024-03-08 10:02:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=160 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c29be1e-fe90-4080-9939-6cff3b538834_isrForCrosstalkSources_855241_39.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c29be1e-fe90-4080-9939-6cff3b538834_isrForCrosstalkSources_855241_39.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66bcd50684f999cdb5fa21c48586f586 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b035e45d-60a3-4173-b84e-28d90f7d2093_isrForCrosstalkSources_855241_53.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b035e45d-60a3-4173-b84e-28d90f7d2093_isrForCrosstalkSources_855241_53.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 079395a087d95f29b3128868bd4d6532 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e0557248-630c-4920-a34f-13340501e317_isrForCrosstalkSources_855238_25.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e0557248-630c-4920-a34f-13340501e317_isrForCrosstalkSources_855238_25.stderr parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01f309efeff832595e99076853af7eed with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/768d4231-35bb-44ff-8b4d-09abfac42442_isrForCrosstalkSources_855231_16.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/768d4231-35bb-44ff-8b4d-09abfac42442_isrForCrosstalkSources_855231_16.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20c28f8c6e3bea61c3c3a54a3b21be3a with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f9a4c428-5457-44ff-b377-d34e3e8f7420_isrForCrosstalkSources_855156_43.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/f9a4c428-5457-44ff-b377-d34e3e8f7420_isrForCrosstalkSources_855156_43.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b16d7441212a19d35f6a4234c9cd0b3e with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/066b5700-44e8-4862-bcaf-1c84a136a142_isrForCrosstalkSources_855234_11.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/066b5700-44e8-4862-bcaf-1c84a136a142_isrForCrosstalkSources_855234_11.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d74c9f358ed6f2f77d9efb5f6f16268 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c3e55535-6b7e-4f79-afca-70d54ff97860_isrForCrosstalkSources_855154_14.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/c3e55535-6b7e-4f79-afca-70d54ff97860_isrForCrosstalkSources_855154_14.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc5a5e376989ddc0427d4a8896f1c8b with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/651a8247-62f0-424d-8989-8cbfb0732223_isrForCrosstalkSources_855233_27.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/651a8247-62f0-424d-8989-8cbfb0732223_isrForCrosstalkSources_855233_27.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1dfa3e5606ba37734414c5de607c7d2 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6ffe80f3-22a0-4321-bb3c-0827b16e0ae2_isrForCrosstalkSources_855238_26.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6ffe80f3-22a0-4321-bb3c-0827b16e0ae2_isrForCrosstalkSources_855238_26.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5f87cbb45e0bacaa5baf3ea97a8886b with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/63e07620-e0a1-48bc-adc6-6419a2e2c6a9_isrForCrosstalkSources_855154_32.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/63e07620-e0a1-48bc-adc6-6419a2e2c6a9_isrForCrosstalkSources_855154_32.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83429793362c26d63e4eaedf0d2a8178 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/29249387-c04b-4323-9913-9782cba152b6_isrForCrosstalkSources_855154_58.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/29249387-c04b-4323-9913-9782cba152b6_isrForCrosstalkSources_855154_58.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 416aa8bbefb1f5ad2203cf1a68675544 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b858442b-0a67-4053-8f56-f54db0311962_isrForCrosstalkSources_855241_51.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b858442b-0a67-4053-8f56-f54db0311962_isrForCrosstalkSources_855241_51.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad1a78892b0cf8fa62009e1cc94c87e with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7b668df0-55f1-4751-8f22-c7efea07f1f2_isrForCrosstalkSources_855232_60.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7b668df0-55f1-4751-8f22-c7efea07f1f2_isrForCrosstalkSources_855232_60.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c95e532c8ca5f11902306bfb76f499f8 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/29d0d8c1-4ace-4c9e-8e43-a7da6383bf78_isrForCrosstalkSources_855239_32.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/29d0d8c1-4ace-4c9e-8e43-a7da6383bf78_isrForCrosstalkSources_855239_32.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a80628ff972b4413c9cbf11df11bcaf with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1198ea1c-20fd-4c7a-aa1e-7d189e2ba82f_isrForCrosstalkSources_855232_34.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1198ea1c-20fd-4c7a-aa1e-7d189e2ba82f_isrForCrosstalkSources_855232_34.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1cf032a1389ee55515c1cc8d30fe18 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e425db60-adf0-4709-9639-18b19ad64f73_isrForCrosstalkSources_855237_8.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e425db60-adf0-4709-9639-18b19ad64f73_isrForCrosstalkSources_855237_8.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd11c6fdf1ad7e6b59c23e7f5abd3d6b with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d5217d6-958c-4e4e-b368-2c79d68d0952_isrForCrosstalkSources_855237_4.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d5217d6-958c-4e4e-b368-2c79d68d0952_isrForCrosstalkSources_855237_4.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 981d890c1b937694b736fa9af8dc0345 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d8313e35-cbd5-4f5d-bd4b-9abce552e0d1_isrForCrosstalkSources_855231_46.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/d8313e35-cbd5-4f5d-bd4b-9abce552e0d1_isrForCrosstalkSources_855231_46.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1ca281d438eb78c4598a501da7f6b31 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/36fa73ea-7b32-4561-a7fc-6ec700360988_isrForCrosstalkSources_855238_34.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/36fa73ea-7b32-4561-a7fc-6ec700360988_isrForCrosstalkSources_855238_34.stderr 2024-03-08 10:02:28 proc_lsst.multi:170 [INFO] [multi] job_id 16914781 proc_lsst.multi INFO: [multi] job_id 16914781 2024-03-08 10:02:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-08 10:02:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914781 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914781 parsl.executors.status_handling DEBUG: Launched block 160 on executor multi with job ID 16914781 parsl.executors.status_handling INFO: Allocated block ID 161 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-161 2024-03-08 10:02:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-161 2024-03-08 10:02:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 104 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 104 2024-03-08 10:02:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=161 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e5377352259da442f3c3362ac882fa9 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/462f0176-78d5-4b0b-a3cf-9a9f62d3f494_isrForCrosstalkSources_855236_45.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/462f0176-78d5-4b0b-a3cf-9a9f62d3f494_isrForCrosstalkSources_855236_45.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a07e7324efca7ef7fb93e1d5bed5362 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12122a40-e26f-4d1d-88ee-29396ce8fb90_isrForCrosstalkSources_855231_56.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12122a40-e26f-4d1d-88ee-29396ce8fb90_isrForCrosstalkSources_855231_56.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7930fb6731ff290e1e35c723d62ac79 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/26530db8-60ce-4127-a8bd-259c2472f33f_isrForCrosstalkSources_855231_10.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/26530db8-60ce-4127-a8bd-259c2472f33f_isrForCrosstalkSources_855231_10.stderr parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e32f893ce801de25ca1b87b5a68060f0 with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6f6591d6-a1b0-4d25-9f44-e42fb69262dd_isrForCrosstalkSources_855233_18.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/6f6591d6-a1b0-4d25-9f44-e42fb69262dd_isrForCrosstalkSources_855233_18.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85e2eedaa15755b57c2b25d48efa690 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6c8e0011-15d5-4355-8ec1-44827284b233_isrForCrosstalkSources_855155_4.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/6c8e0011-15d5-4355-8ec1-44827284b233_isrForCrosstalkSources_855155_4.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c64dceadd562fb6b74d3b9b1e5587de with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/df032206-a1cf-45e0-8d03-396f825835c5_isrForCrosstalkSources_855155_28.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/df032206-a1cf-45e0-8d03-396f825835c5_isrForCrosstalkSources_855155_28.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94c74c167490d3db9e966421930abf73 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2f3b7344-0c61-4ddf-9664-eb92bff275a4_isrForCrosstalkSources_855239_2.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2f3b7344-0c61-4ddf-9664-eb92bff275a4_isrForCrosstalkSources_855239_2.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f0dc00a0b55e39b86485cd48b33ce72 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/acfc4072-4f17-4d38-9c17-67df147e79ed_isrForCrosstalkSources_855154_23.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/acfc4072-4f17-4d38-9c17-67df147e79ed_isrForCrosstalkSources_855154_23.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea89b067982c97862bff7e16fb0d1ee with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/57c0f898-4fbf-4c35-84b5-f5a9446da2e8_isrForCrosstalkSources_855238_57.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/57c0f898-4fbf-4c35-84b5-f5a9446da2e8_isrForCrosstalkSources_855238_57.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ba49c52d385ae4907acdd2d9177fcb5 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/430d8c6e-98c2-4db0-8a93-cfe983f5b6e5_isrForCrosstalkSources_855233_51.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/430d8c6e-98c2-4db0-8a93-cfe983f5b6e5_isrForCrosstalkSources_855233_51.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f6baede5c6b7a1ca728ea8de3a21f16 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/acf94272-d692-422d-939f-f4f2ca46c2b1_isrForCrosstalkSources_855239_44.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/acf94272-d692-422d-939f-f4f2ca46c2b1_isrForCrosstalkSources_855239_44.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7826818c27328e1941eedcab5fbcc16 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/82d10652-90a8-4395-8e6d-cf943278eff4_isrForCrosstalkSources_855232_33.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/82d10652-90a8-4395-8e6d-cf943278eff4_isrForCrosstalkSources_855232_33.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5e3f27321249db334f316d8d2db3493 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c926047-dbea-49d0-87ed-626f0c616e6f_isrForCrosstalkSources_855154_3.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c926047-dbea-49d0-87ed-626f0c616e6f_isrForCrosstalkSources_855154_3.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdb2ed50c8bec6552708ec8b0b6417cc with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a91b831-ff84-4498-8113-504373243729_isrForCrosstalkSources_855235_39.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5a91b831-ff84-4498-8113-504373243729_isrForCrosstalkSources_855235_39.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a8f8c7ef1411ab7572c7054593398ed with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/93b62b73-079f-465d-af84-3893af5f48d0_isrForCrosstalkSources_855154_16.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/93b62b73-079f-465d-af84-3893af5f48d0_isrForCrosstalkSources_855154_16.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b482cd2ed6ac48358014b1bf5d4144eb with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b23d8cb0-7b14-4462-b06a-35d5cd54c946_isrForCrosstalkSources_855239_31.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/b23d8cb0-7b14-4462-b06a-35d5cd54c946_isrForCrosstalkSources_855239_31.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dc78f05f1488e1db4aa8e6d1a6cf5b0 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ce7f644-d681-4d2e-ae3a-23dfebf79afc_isrForCrosstalkSources_855236_24.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5ce7f644-d681-4d2e-ae3a-23dfebf79afc_isrForCrosstalkSources_855236_24.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81b2cc98c3697e52a67180c5dc58ce1e with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67f7d643-62d2-49e4-a0b4-5b218e04d57a_isrForCrosstalkSources_855239_51.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67f7d643-62d2-49e4-a0b4-5b218e04d57a_isrForCrosstalkSources_855239_51.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34ce34a862fc9fbc5cbb8cc4d07bac9f with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ddd3a7da-49a4-48fe-ab01-e6dbd6b82485_isrForCrosstalkSources_855233_5.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/ddd3a7da-49a4-48fe-ab01-e6dbd6b82485_isrForCrosstalkSources_855233_5.stderr parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 545a61d86f11a2d3513ed82a65d3110d with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6496040-4ac9-42e2-b4d6-174336ebcd3c_isrForCrosstalkSources_855240_5.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d6496040-4ac9-42e2-b4d6-174336ebcd3c_isrForCrosstalkSources_855240_5.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb104b6931494adb97f0d1588376189e with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/95dc7636-ec93-4419-a4ab-aac6ef153337_isrForCrosstalkSources_855155_19.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/95dc7636-ec93-4419-a4ab-aac6ef153337_isrForCrosstalkSources_855155_19.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16bedec17181f32f170bdcf1f46c68e with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03acbb5e-90ca-49c1-b1fb-864c57f9def9_isrForCrosstalkSources_855238_15.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/03acbb5e-90ca-49c1-b1fb-864c57f9def9_isrForCrosstalkSources_855238_15.stderr parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0ddd1c7fda423ebe070ec809e8a27e4 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e446e872-2fa8-4c81-8c1f-f9e2e43a80ca_isrForCrosstalkSources_855241_46.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/e446e872-2fa8-4c81-8c1f-f9e2e43a80ca_isrForCrosstalkSources_855241_46.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d215fd7692feabf79689efde80aca0 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/6b0640be-eb69-475c-811f-62a1a2d7f0fa_isrForCrosstalkSources_855237_5.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/6b0640be-eb69-475c-811f-62a1a2d7f0fa_isrForCrosstalkSources_855237_5.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85b94a8301cda730691243f7ea9df6b7 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/278400eb-6ea6-4e75-8653-886c9fb51914_isrForCrosstalkSources_855238_12.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/278400eb-6ea6-4e75-8653-886c9fb51914_isrForCrosstalkSources_855238_12.stderr parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db8281b478c5b45777534d85a64cc66d with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f8d184cb-4444-413f-b5cf-1526d681a27b_isrForCrosstalkSources_855233_42.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f8d184cb-4444-413f-b5cf-1526d681a27b_isrForCrosstalkSources_855233_42.stderr parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 166a024268d2c7e389be5e7f1e844553 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/200953d2-f80d-4c6a-9f96-3031e76259ac_isrForCrosstalkSources_855232_31.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/200953d2-f80d-4c6a-9f96-3031e76259ac_isrForCrosstalkSources_855232_31.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73b582df515419cf47330253e854e387 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/377c5e19-1101-40b0-9e9a-90b29f167ebb_isrForCrosstalkSources_855242_3.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/377c5e19-1101-40b0-9e9a-90b29f167ebb_isrForCrosstalkSources_855242_3.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e46ac2a009c0400ecdc58d1b1cafddc with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67be035e-7f76-4e7e-bc10-a85a0e191841_isrForCrosstalkSources_855239_6.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/67be035e-7f76-4e7e-bc10-a85a0e191841_isrForCrosstalkSources_855239_6.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b64264577caed548cf029811815df8e2 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e8ae09dd-8e35-4fe7-9926-f2bda87bccea_isrForCrosstalkSources_855238_35.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e8ae09dd-8e35-4fe7-9926-f2bda87bccea_isrForCrosstalkSources_855238_35.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b25d53defc5a36a4343b91911051dc4 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/2c697529-4393-410c-bdd3-3c2a32c593b8_isrForCrosstalkSources_855233_6.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/2c697529-4393-410c-bdd3-3c2a32c593b8_isrForCrosstalkSources_855233_6.stderr parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4a0eeb6a1b1002f4d2c8e7b61c4d6b0 with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/23ae707c-f85d-4644-9558-5c6d46e8c223_isrForCrosstalkSources_855231_32.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/23ae707c-f85d-4644-9558-5c6d46e8c223_isrForCrosstalkSources_855231_32.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 615227831548f13fa2c749fd2098863b with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ceee6a31-aa30-4e15-b297-68abfbc5f334_isrForCrosstalkSources_855242_1.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ceee6a31-aa30-4e15-b297-68abfbc5f334_isrForCrosstalkSources_855242_1.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1e6ff5ee0c6b8ed4262810e0b66e1be with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/1f55af03-0b6d-41ae-a571-ca791ccb0f8e_isrForCrosstalkSources_855234_49.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/1f55af03-0b6d-41ae-a571-ca791ccb0f8e_isrForCrosstalkSources_855234_49.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95977fd13a9dd690690167bfdf1af44a with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/1d4292f4-4eea-4fe3-8858-bd06614e0d56_isrForCrosstalkSources_855154_61.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/1d4292f4-4eea-4fe3-8858-bd06614e0d56_isrForCrosstalkSources_855154_61.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0bf3ba941f11edb9355a8e1ef0e2640 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f412f460-1c47-4c55-a31a-f36069d6b9c4_isrForCrosstalkSources_855236_57.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f412f460-1c47-4c55-a31a-f36069d6b9c4_isrForCrosstalkSources_855236_57.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbc03fcf768911d47de76991cb3c9a62 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d397ee31-d81a-494a-b3b4-a99e5c5c61c0_isrForCrosstalkSources_855241_38.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d397ee31-d81a-494a-b3b4-a99e5c5c61c0_isrForCrosstalkSources_855241_38.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 719520c749914d092cc12ba777ca280a with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/31f78770-896e-4801-a35c-7af5fde1a431_isrForCrosstalkSources_855238_42.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/31f78770-896e-4801-a35c-7af5fde1a431_isrForCrosstalkSources_855238_42.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68a7c10c3674d3c1f449e1c8de4bd9a4 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80e166a7-2fe9-46bb-9dbe-e3eca0016eaa_isrForCrosstalkSources_855235_55.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80e166a7-2fe9-46bb-9dbe-e3eca0016eaa_isrForCrosstalkSources_855235_55.stderr parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5ea3b2a7ce53564ac191dc262a22383 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d965b5b5-6788-442e-95c2-4cd8fe820317_isrForCrosstalkSources_855240_50.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d965b5b5-6788-442e-95c2-4cd8fe820317_isrForCrosstalkSources_855240_50.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0970026f2bd91337bb84409de1be4410 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/cad3f9e6-82d3-4fe0-8ea8-cb7482394d51_isrForCrosstalkSources_855156_49.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/cad3f9e6-82d3-4fe0-8ea8-cb7482394d51_isrForCrosstalkSources_855156_49.stderr parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aae384899eaad66612ebc97f867b744 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/37dd618e-7ae9-4a02-9d78-1872447be4d6_isrForCrosstalkSources_855236_46.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/37dd618e-7ae9-4a02-9d78-1872447be4d6_isrForCrosstalkSources_855236_46.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da9d12d724eff66c751cdff58a083585 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/503274f4-3164-4df3-9df3-670eba3a8d34_isrForCrosstalkSources_855232_59.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/503274f4-3164-4df3-9df3-670eba3a8d34_isrForCrosstalkSources_855232_59.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaa55a2d9d1676f2a7c581a30de462f5 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/abcc1e3b-62fa-4e0a-a154-a3537474f409_isrForCrosstalkSources_855241_37.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/abcc1e3b-62fa-4e0a-a154-a3537474f409_isrForCrosstalkSources_855241_37.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 948a330f476e94869d7f5c28f63f6d4c with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/e67bc7ed-e7b6-48d9-9ee6-d085b540c52e_isrForCrosstalkSources_855234_38.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/e67bc7ed-e7b6-48d9-9ee6-d085b540c52e_isrForCrosstalkSources_855234_38.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d3c47cf6d8c33f9cb331edc1ab59ab8 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/efe6dc1c-dedf-420e-a371-c3cfbfa8b417_isrForCrosstalkSources_855237_9.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/efe6dc1c-dedf-420e-a371-c3cfbfa8b417_isrForCrosstalkSources_855237_9.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d987fec580ea6744d9da2cdbe3aa03f with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/78e62b4c-4a7a-4811-bcd7-2b21d2458b88_isrForCrosstalkSources_855242_53.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/78e62b4c-4a7a-4811-bcd7-2b21d2458b88_isrForCrosstalkSources_855242_53.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c9401bde0ed0f743bdb71024714caf with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ac1b67f8-6b3b-42c0-861a-f493de4be69e_isrForCrosstalkSources_855238_14.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ac1b67f8-6b3b-42c0-861a-f493de4be69e_isrForCrosstalkSources_855238_14.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16c122a1dcff21fd181ff82ea868faf with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0f90e501-fc04-4d6e-88f8-87832ce6c969_isrForCrosstalkSources_855232_22.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0f90e501-fc04-4d6e-88f8-87832ce6c969_isrForCrosstalkSources_855232_22.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec3507207ed14e681b226deacc2fb9f6 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/075e741f-d06b-4eb2-85da-f79cfeab1736_isrForCrosstalkSources_855233_46.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/075e741f-d06b-4eb2-85da-f79cfeab1736_isrForCrosstalkSources_855233_46.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f239d9ea553273611771d25aaed7b41 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ec170161-85b1-488c-80f4-6d98dde8c795_isrForCrosstalkSources_855239_21.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/ec170161-85b1-488c-80f4-6d98dde8c795_isrForCrosstalkSources_855239_21.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95d3d9ab85d996591a2c69d92e73332d with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/aa865f4c-f897-415e-895c-8908315f08c3_isrForCrosstalkSources_855237_30.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/aa865f4c-f897-415e-895c-8908315f08c3_isrForCrosstalkSources_855237_30.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d14dacf30379b4e4484eb2afee39b6 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0bf31726-8b71-4a80-baa4-a2eeef67a8ce_isrForCrosstalkSources_855155_46.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0bf31726-8b71-4a80-baa4-a2eeef67a8ce_isrForCrosstalkSources_855155_46.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f00e0dfb3eb84a8f9c4730a40c24b3ce with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6699e11d-e6e5-48d7-a6de-b290481fe60f_isrForCrosstalkSources_855232_7.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6699e11d-e6e5-48d7-a6de-b290481fe60f_isrForCrosstalkSources_855232_7.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2855acbd34c2f6a026c6db153a3bf2d5 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/62a87044-4d65-4f2f-a595-d927c1687340_isrForCrosstalkSources_855241_4.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/62a87044-4d65-4f2f-a595-d927c1687340_isrForCrosstalkSources_855241_4.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ff072c773581f469d2777ba20d8b879 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1545c96f-f996-4871-8053-2fe9d49fb965_isrForCrosstalkSources_855155_23.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/1545c96f-f996-4871-8053-2fe9d49fb965_isrForCrosstalkSources_855155_23.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7e4c3c02e0ed0363304504260da6e90 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c5035635-6d99-4e1d-a1c9-5364cff0b3d7_isrForCrosstalkSources_855239_10.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c5035635-6d99-4e1d-a1c9-5364cff0b3d7_isrForCrosstalkSources_855239_10.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 965385e39437fd5d26ca82b1d4e71037 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12e39556-d7ab-43fc-95d2-35b7e7611429_isrForCrosstalkSources_855231_58.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/12e39556-d7ab-43fc-95d2-35b7e7611429_isrForCrosstalkSources_855231_58.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d7968ed112588c04282e1ab47cdb29 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/085896b7-f9ff-4ab3-be86-6c75cf88fe63_isrForCrosstalkSources_855231_49.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/085896b7-f9ff-4ab3-be86-6c75cf88fe63_isrForCrosstalkSources_855231_49.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6294cc3fbd89966107c49ecb4d8b56ee with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e316369f-f0b6-4a02-8ee0-e96e459a83bb_isrForCrosstalkSources_855238_59.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e316369f-f0b6-4a02-8ee0-e96e459a83bb_isrForCrosstalkSources_855238_59.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a123393d7eb2b86d2c3f74fca0d75afe with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5c54f4ae-f7d3-44ec-b0ee-ecf36b3e3570_isrForCrosstalkSources_855242_32.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5c54f4ae-f7d3-44ec-b0ee-ecf36b3e3570_isrForCrosstalkSources_855242_32.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365b5bf82a302ca08912b7527210f31b with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4924b476-7e29-4345-9c07-45f811f519ba_isrForCrosstalkSources_855238_60.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4924b476-7e29-4345-9c07-45f811f519ba_isrForCrosstalkSources_855238_60.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c42829037a408c20ccd5500568b9e696 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bf4deac-acc2-4221-8fdc-f0d766de0b04_isrForCrosstalkSources_855231_42.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4bf4deac-acc2-4221-8fdc-f0d766de0b04_isrForCrosstalkSources_855231_42.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12a9c132f7738c42e1ad4e7478b8cea2 with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ff12bda2-fc6b-4c16-97f8-0e1ada968367_isrForCrosstalkSources_855236_7.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/ff12bda2-fc6b-4c16-97f8-0e1ada968367_isrForCrosstalkSources_855236_7.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c75f95a840d68541e9bff79ac166b649 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8c35c126-2ff8-4f35-8276-2ef041660b03_isrForCrosstalkSources_855238_48.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8c35c126-2ff8-4f35-8276-2ef041660b03_isrForCrosstalkSources_855238_48.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd4f524c796d58557408294eb094b7cf with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f706eda-7ed3-4736-9e34-e63986662329_isrForCrosstalkSources_855241_5.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0f706eda-7ed3-4736-9e34-e63986662329_isrForCrosstalkSources_855241_5.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7ef8631b084fe37f9e0317685a47b25 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1837c797-20c3-45c4-b4dd-ff7124b91ec8_isrForCrosstalkSources_855238_23.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1837c797-20c3-45c4-b4dd-ff7124b91ec8_isrForCrosstalkSources_855238_23.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2df619f2e5a42a7220dfab627b531cb0 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/292a5b69-0cdf-49a4-84c7-5f2f82b34fc6_isrForCrosstalkSources_855231_50.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/292a5b69-0cdf-49a4-84c7-5f2f82b34fc6_isrForCrosstalkSources_855231_50.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49316469af29882588e2e211e039db61 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d7698ada-bb2f-41e1-92c1-0e3ed85cd5b3_isrForCrosstalkSources_855238_17.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d7698ada-bb2f-41e1-92c1-0e3ed85cd5b3_isrForCrosstalkSources_855238_17.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20ca163db3e3480db073b390a632bb98 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8b1e9eb6-5543-4cff-8cce-2da64c8df4eb_isrForCrosstalkSources_855234_57.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/8b1e9eb6-5543-4cff-8cce-2da64c8df4eb_isrForCrosstalkSources_855234_57.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a97afbdc2d2c9951aabf3411d19d3455 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9c51466c-d80e-44d4-8a4c-dc6d2657d538_isrForCrosstalkSources_855239_1.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9c51466c-d80e-44d4-8a4c-dc6d2657d538_isrForCrosstalkSources_855239_1.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6faae81855009eefcfd3dbb7cd651f4 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3fcfc5b-1599-42c6-84b6-aee432815de0_isrForCrosstalkSources_855236_61.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3fcfc5b-1599-42c6-84b6-aee432815de0_isrForCrosstalkSources_855236_61.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cccc42186cd22d71833080d4dd577919 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/924a3291-486f-4f4b-8503-b577f738929e_isrForCrosstalkSources_855154_8.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/924a3291-486f-4f4b-8503-b577f738929e_isrForCrosstalkSources_855154_8.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3bd4e1f82d7c32c232e82b96b5df499 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bef3cfb8-8490-4a71-b7dd-65d966c2492d_isrForCrosstalkSources_855236_41.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bef3cfb8-8490-4a71-b7dd-65d966c2492d_isrForCrosstalkSources_855236_41.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5259473110f9c5dbc0bd12f96b3073f3 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8b8f4273-99ef-42f5-a51e-e48a23f7a445_isrForCrosstalkSources_855232_52.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/8b8f4273-99ef-42f5-a51e-e48a23f7a445_isrForCrosstalkSources_855232_52.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b2a1ecd37a410a447511888d69ad4e0 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/bb758d61-a15a-40bc-a334-c783c326070c_isrForCrosstalkSources_855154_56.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/bb758d61-a15a-40bc-a334-c783c326070c_isrForCrosstalkSources_855154_56.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfb797e2afacd72a8994b70eae37e248 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/214319ec-cdaa-428d-922f-29b4ad933835_isrForCrosstalkSources_855155_61.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/214319ec-cdaa-428d-922f-29b4ad933835_isrForCrosstalkSources_855155_61.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d53fc8698b80bea9c3954e56b52d5a7b with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/979b0487-34fa-4bf8-8f31-5f421eb23c08_isrForCrosstalkSources_855234_54.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/979b0487-34fa-4bf8-8f31-5f421eb23c08_isrForCrosstalkSources_855234_54.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e36495ac2c6777e7be4ffcc86efc4eab with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/29d7ade1-5731-4a6e-bba1-c87d045a106e_isrForCrosstalkSources_855156_52.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/29d7ade1-5731-4a6e-bba1-c87d045a106e_isrForCrosstalkSources_855156_52.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d93449bac226bcc5f997c50a0f95d91 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/83302c21-39fc-4574-81bc-df36d4702135_isrForCrosstalkSources_855154_41.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/83302c21-39fc-4574-81bc-df36d4702135_isrForCrosstalkSources_855154_41.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a450114b660d40daf82082f625d6d1 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/11131618-c2cf-4853-a928-396c4b09014b_isrForCrosstalkSources_855233_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/11131618-c2cf-4853-a928-396c4b09014b_isrForCrosstalkSources_855233_60.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4067d9df4ab0f8e05480d798dcb61a51 with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/29a74a82-4cb5-4b44-ba11-47c0321649cb_isrForCrosstalkSources_855242_57.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/29a74a82-4cb5-4b44-ba11-47c0321649cb_isrForCrosstalkSources_855242_57.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3028c70a77f781c36b6f3d6a4dd6fa33 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3398ed4b-1737-44bc-a84c-096b13329720_isrForCrosstalkSources_855233_49.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/3398ed4b-1737-44bc-a84c-096b13329720_isrForCrosstalkSources_855233_49.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecc47e1d2f18d8edd33257389cff3a36 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/63abfa82-775c-4c17-9286-5c95ce1dac4f_isrForCrosstalkSources_855239_36.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/63abfa82-775c-4c17-9286-5c95ce1dac4f_isrForCrosstalkSources_855239_36.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6197e308d31396cd89467956c75b744 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c0c04287-5319-46a4-b465-03a938526f95_isrForCrosstalkSources_855156_26.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c0c04287-5319-46a4-b465-03a938526f95_isrForCrosstalkSources_855156_26.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 528c3efb11ae2a8154e8fb36073225db with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ee8c9e38-a41b-4a54-b3fb-fcd5e96c6a2f_isrForCrosstalkSources_855240_60.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ee8c9e38-a41b-4a54-b3fb-fcd5e96c6a2f_isrForCrosstalkSources_855240_60.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b7cf3b33e6f69037a2471fc729bd78f with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7facc6ce-81ba-483e-bdc6-60aeb010a8cb_isrForCrosstalkSources_855241_1.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7facc6ce-81ba-483e-bdc6-60aeb010a8cb_isrForCrosstalkSources_855241_1.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a32ec297ba35bd52a8af4adfca39e546 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/323c6cff-ca98-48cc-97c4-263144c4b466_isrForCrosstalkSources_855238_40.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/323c6cff-ca98-48cc-97c4-263144c4b466_isrForCrosstalkSources_855238_40.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a243f25ffb5bf7c62a5181f2f3213bd8 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/55767d2b-33ee-44c7-b24e-d11487d92484_isrForCrosstalkSources_855234_25.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/55767d2b-33ee-44c7-b24e-d11487d92484_isrForCrosstalkSources_855234_25.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be18a58d831bad01e31a046ea1530548 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/82bcd5f1-e3f5-43d5-9bb6-00454056dc11_isrForCrosstalkSources_855241_31.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/82bcd5f1-e3f5-43d5-9bb6-00454056dc11_isrForCrosstalkSources_855241_31.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdfdde20bf1c475347735a41c5786f22 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a8a0a5bf-83c2-47e9-baa0-1f53ad591362_isrForCrosstalkSources_855237_31.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a8a0a5bf-83c2-47e9-baa0-1f53ad591362_isrForCrosstalkSources_855237_31.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05bf4c004c8a6449e0d84100f071fae2 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/caa2711f-4821-4f08-b8ed-0f013fa59262_isrForCrosstalkSources_855232_39.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/caa2711f-4821-4f08-b8ed-0f013fa59262_isrForCrosstalkSources_855232_39.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1398ff9a129c8cfab348c5f7d9337d2 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4c7f9581-27e9-4ec7-b534-8e72d263ac8c_isrForCrosstalkSources_855156_42.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4c7f9581-27e9-4ec7-b534-8e72d263ac8c_isrForCrosstalkSources_855156_42.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4995d530a7009e17f613c9d4a3b61f with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/83fbbf08-3e43-45e4-a019-c8f4e9a4f39e_isrForCrosstalkSources_855234_7.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/83fbbf08-3e43-45e4-a019-c8f4e9a4f39e_isrForCrosstalkSources_855234_7.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f82c236aac6dfe27c68fd0c01fbc87 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/547c98fe-ac5b-442b-a23e-e8ecf8745f2b_isrForCrosstalkSources_855242_10.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/547c98fe-ac5b-442b-a23e-e8ecf8745f2b_isrForCrosstalkSources_855242_10.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43e300156604297aeca33e22d3f58384 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/41d81427-5004-4240-8d95-d3e7525c5687_isrForCrosstalkSources_855240_57.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/41d81427-5004-4240-8d95-d3e7525c5687_isrForCrosstalkSources_855240_57.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb632ded6b7509e1a17fa92227b7e0ae with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/01203b56-0f1a-4fe1-988e-4d8f9532918c_isrForCrosstalkSources_855237_57.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/01203b56-0f1a-4fe1-988e-4d8f9532918c_isrForCrosstalkSources_855237_57.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e70e5a7f1f8d61ebffe55a68527e575 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c8dd9e81-1b43-451d-b4e0-9097d653f77d_isrForCrosstalkSources_855232_21.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c8dd9e81-1b43-451d-b4e0-9097d653f77d_isrForCrosstalkSources_855232_21.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29fd90b164db26dfeba4fa87e4c32d6b with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d2f344e-996e-4ea4-b139-47482055f379_isrForCrosstalkSources_855237_50.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/0d2f344e-996e-4ea4-b139-47482055f379_isrForCrosstalkSources_855237_50.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9cd532a1d470aeb5c18bd3864e9f4d with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24acabc7-98ff-489b-8d09-63d1f51143bb_isrForCrosstalkSources_855241_41.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24acabc7-98ff-489b-8d09-63d1f51143bb_isrForCrosstalkSources_855241_41.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e380508ffee4e2eb93877cc93227403 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2f117df-4f7b-4d83-a223-8711d6704c55_isrForCrosstalkSources_855155_29.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2f117df-4f7b-4d83-a223-8711d6704c55_isrForCrosstalkSources_855155_29.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 917805c8298bc858d9df1be483019a90 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/95d2d6a3-9f9a-4a27-8d10-4b7d81948f44_isrForCrosstalkSources_855235_36.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/95d2d6a3-9f9a-4a27-8d10-4b7d81948f44_isrForCrosstalkSources_855235_36.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e613d9161dc109daa4cf2159a3338a9 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/50dc01e8-7fc6-4e92-b899-dc4a154e63ee_isrForCrosstalkSources_855237_33.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/50dc01e8-7fc6-4e92-b899-dc4a154e63ee_isrForCrosstalkSources_855237_33.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33d6fac2a2796cfb4eecfe9891bbfea9 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7e8043d4-832d-4016-97e1-ce48a7ad47c6_isrForCrosstalkSources_855236_53.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/7e8043d4-832d-4016-97e1-ce48a7ad47c6_isrForCrosstalkSources_855236_53.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d844dc6f28d63c8aed7dc8ef55c7058 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/7be63ae7-90e0-4cd7-9bc0-fc25f62469bf_isrForCrosstalkSources_855156_39.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/7be63ae7-90e0-4cd7-9bc0-fc25f62469bf_isrForCrosstalkSources_855156_39.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd67097d67dda63d8f8e104caa7038fd with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/602db23c-d2eb-428f-b6c2-9bf5009aa087_isrForCrosstalkSources_855232_19.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/602db23c-d2eb-428f-b6c2-9bf5009aa087_isrForCrosstalkSources_855232_19.stderr 2024-03-08 10:02:32 proc_lsst.multi:170 [INFO] [multi] job_id 16914782 proc_lsst.multi INFO: [multi] job_id 16914782 2024-03-08 10:02:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-08 10:02:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914782 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914782 parsl.executors.status_handling DEBUG: Launched block 161 on executor multi with job ID 16914782 parsl.executors.status_handling INFO: Allocated block ID 162 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-162 2024-03-08 10:02:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-162 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 105 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 105 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=162 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc0e3dd335b37e5d93ec5a9ee0b4bbc2 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3ef22ab7-cac7-4dbd-942c-060b34edcf5b_isrForCrosstalkSources_855155_43.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3ef22ab7-cac7-4dbd-942c-060b34edcf5b_isrForCrosstalkSources_855155_43.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540df451f3d1867d77fea5a2304afa2a with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/b0a1949a-ab4b-4934-916b-230ec6f3bb9c_isrForCrosstalkSources_855240_32.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/b0a1949a-ab4b-4934-916b-230ec6f3bb9c_isrForCrosstalkSources_855240_32.stderr 2024-03-08 10:02:32 proc_lsst.multi:170 [INFO] [multi] job_id 16914783 proc_lsst.multi INFO: [multi] job_id 16914783 2024-03-08 10:02:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-08 10:02:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914783 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914783 parsl.executors.status_handling DEBUG: Launched block 162 on executor multi with job ID 16914783 parsl.executors.status_handling INFO: Allocated block ID 163 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-163 2024-03-08 10:02:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-163 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 106 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 106 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=163 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44eff985df6011a6774e79a41a3417c7 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14b958ac-0a33-4cca-8fd8-e56ccb105eab_isrForCrosstalkSources_855240_6.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/14b958ac-0a33-4cca-8fd8-e56ccb105eab_isrForCrosstalkSources_855240_6.stderr 2024-03-08 10:02:32 proc_lsst.multi:170 [INFO] [multi] job_id 16914784 proc_lsst.multi INFO: [multi] job_id 16914784 2024-03-08 10:02:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-08 10:02:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914784 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914784 parsl.executors.status_handling DEBUG: Launched block 163 on executor multi with job ID 16914784 parsl.executors.status_handling INFO: Allocated block ID 164 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-164 2024-03-08 10:02:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-164 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 107 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 107 2024-03-08 10:02:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=164 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb58817a903bd2b87212936cad8cff5f with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/353f74cd-3acf-491d-beb3-5128434c1419_isrForCrosstalkSources_855155_55.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/353f74cd-3acf-491d-beb3-5128434c1419_isrForCrosstalkSources_855155_55.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94bed6aae596adcc455ffd165a51708e with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/fdf6e0de-23ec-4400-8ee1-188a3f85fdcd_isrForCrosstalkSources_855155_56.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/fdf6e0de-23ec-4400-8ee1-188a3f85fdcd_isrForCrosstalkSources_855155_56.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc1ef5cf661af0fd40e2cdc037f0ee1 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/aec6ba88-e7e3-4b77-9630-fb5f9270391d_isrForCrosstalkSources_855236_1.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/aec6ba88-e7e3-4b77-9630-fb5f9270391d_isrForCrosstalkSources_855236_1.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4662ea86d7ce7e81d4b1133d4bb57f78 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/97c8c4de-fe22-455e-abee-06771105220c_isrForCrosstalkSources_855238_21.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/97c8c4de-fe22-455e-abee-06771105220c_isrForCrosstalkSources_855238_21.stderr parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a20be48e62305df690d6dc6b07bbee93 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d106770b-54a6-4991-aa24-99b11a992860_isrForCrosstalkSources_855238_50.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d106770b-54a6-4991-aa24-99b11a992860_isrForCrosstalkSources_855238_50.stderr 2024-03-08 10:02:33 proc_lsst.multi:170 [INFO] [multi] job_id 16914785 proc_lsst.multi INFO: [multi] job_id 16914785 2024-03-08 10:02:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-08 10:02:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914785 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914785 parsl.executors.status_handling DEBUG: Launched block 164 on executor multi with job ID 16914785 parsl.executors.status_handling INFO: Allocated block ID 165 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-165 2024-03-08 10:02:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-165 2024-03-08 10:02:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 108 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 108 2024-03-08 10:02:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=165 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:34 proc_lsst.multi:170 [INFO] [multi] job_id 16914786 proc_lsst.multi INFO: [multi] job_id 16914786 2024-03-08 10:02:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-08 10:02:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914786 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914786 parsl.executors.status_handling DEBUG: Launched block 165 on executor multi with job ID 16914786 parsl.executors.status_handling INFO: Allocated block ID 166 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-166 2024-03-08 10:02:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-166 2024-03-08 10:02:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 109 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 109 2024-03-08 10:02:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=166 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0daf87b77f7840cbf10ee49533d23c8c with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a22f498c-694d-4663-9bca-dcaff74a22c8_isrForCrosstalkSources_855232_61.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/a22f498c-694d-4663-9bca-dcaff74a22c8_isrForCrosstalkSources_855232_61.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32f85ad5caf9009eb3fb3e184b5ce83c with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4e275644-e9b9-43b2-b676-5b8ce77e7fcd_isrForCrosstalkSources_855236_32.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4e275644-e9b9-43b2-b676-5b8ce77e7fcd_isrForCrosstalkSources_855236_32.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43e7e456965e653637b377c80d6ea622 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ef6b9856-c8ef-42b6-83fc-36a1f7797735_isrForCrosstalkSources_855234_5.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ef6b9856-c8ef-42b6-83fc-36a1f7797735_isrForCrosstalkSources_855234_5.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a976fdb4412b1a61e4e326111f27ffd with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72c2908c-4dcc-4cd8-b08b-6f4d02a33e7c_isrForCrosstalkSources_855240_29.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72c2908c-4dcc-4cd8-b08b-6f4d02a33e7c_isrForCrosstalkSources_855240_29.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc6740005efa3a23751fe970dea1f190 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1312ce44-4355-4338-afe9-6f4c073fe3a0_isrForCrosstalkSources_855238_3.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/1312ce44-4355-4338-afe9-6f4c073fe3a0_isrForCrosstalkSources_855238_3.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 161ab555fc799a11fb87f23002ce2a30 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c49cfd3-d62d-4789-a4e7-44e7196f10bf_isrForCrosstalkSources_855154_60.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4c49cfd3-d62d-4789-a4e7-44e7196f10bf_isrForCrosstalkSources_855154_60.stderr 2024-03-08 10:02:35 proc_lsst.multi:170 [INFO] [multi] job_id 16914787 proc_lsst.multi INFO: [multi] job_id 16914787 2024-03-08 10:02:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-08 10:02:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914787 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914787 parsl.executors.status_handling DEBUG: Launched block 166 on executor multi with job ID 16914787 parsl.executors.status_handling INFO: Allocated block ID 167 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-167 2024-03-08 10:02:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-167 2024-03-08 10:02:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 110 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 110 2024-03-08 10:02:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=167 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0fd8d5857e907184ab6b1f9edb2387e with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/04ee46fa-d488-48ac-86ae-e72c56d3f80a_isrForCrosstalkSources_855232_43.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/04ee46fa-d488-48ac-86ae-e72c56d3f80a_isrForCrosstalkSources_855232_43.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbfe5250204227bfe09a569d50a4e3e6 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/854c85c2-aba8-47ef-8e09-24395831b681_isrForCrosstalkSources_855155_41.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/854c85c2-aba8-47ef-8e09-24395831b681_isrForCrosstalkSources_855155_41.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09c5d353427595c74527baeaeb2c6fc4 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/13df86d2-0da7-4c09-85fd-7ba504492524_isrForCrosstalkSources_855233_4.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/13df86d2-0da7-4c09-85fd-7ba504492524_isrForCrosstalkSources_855233_4.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41c412ae67bf75f50540c67591eade13 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/357a1cf2-7853-417c-bc6e-419ad7a91221_isrForCrosstalkSources_855155_7.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/357a1cf2-7853-417c-bc6e-419ad7a91221_isrForCrosstalkSources_855155_7.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49cb848265723b6923082d846799c715 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4aa88803-8a03-4cb1-a7d4-88120eaf2093_isrForCrosstalkSources_855238_24.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/4aa88803-8a03-4cb1-a7d4-88120eaf2093_isrForCrosstalkSources_855238_24.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0b5d414b69202ce0d1b3cb743a04bcf with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0762d972-5188-4037-a42e-c0e1010e5bc7_isrForCrosstalkSources_855234_8.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/0762d972-5188-4037-a42e-c0e1010e5bc7_isrForCrosstalkSources_855234_8.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5e4e8ddbec331f26879f9db535820ca with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6beae20c-d540-40ff-b902-68ccda0f55b6_isrForCrosstalkSources_855238_33.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6beae20c-d540-40ff-b902-68ccda0f55b6_isrForCrosstalkSources_855238_33.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ee74a571aed20e53da38a9a155e4116 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8a1e347-999b-4131-bc91-962ff79ecbcd_isrForCrosstalkSources_855234_28.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8a1e347-999b-4131-bc91-962ff79ecbcd_isrForCrosstalkSources_855234_28.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd879881296f63bf1ea9786252aa5ef with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8ea27f4-5d72-4052-b2a7-72b85e343abc_isrForCrosstalkSources_855234_46.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a8ea27f4-5d72-4052-b2a7-72b85e343abc_isrForCrosstalkSources_855234_46.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efcbf9fddbd9326951c3a3b139d1df6d with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/944afd90-3d21-431b-9324-6cf5f07a649c_isrForCrosstalkSources_855155_18.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/944afd90-3d21-431b-9324-6cf5f07a649c_isrForCrosstalkSources_855155_18.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e392558b5dc4fcfcb6ef54b1cd1f1d4 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0ca8f778-37c9-4c91-84b3-3a5c778c4988_isrForCrosstalkSources_855232_1.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/0ca8f778-37c9-4c91-84b3-3a5c778c4988_isrForCrosstalkSources_855232_1.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 565d6d532c0269af99a70d3dc58ce9ba with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/984f13c8-842f-4e30-879a-04e1529e7456_isrForCrosstalkSources_855235_32.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/984f13c8-842f-4e30-879a-04e1529e7456_isrForCrosstalkSources_855235_32.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3adb254a6a8ed9388ca66af5953024d with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2a81fde4-c219-4ba3-a85a-a345b5898854_isrForCrosstalkSources_855155_17.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/2a81fde4-c219-4ba3-a85a-a345b5898854_isrForCrosstalkSources_855155_17.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48b31f885f5fc61bf57fd4c7278f5e3f with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a5223054-fdf3-43ea-9ea4-c5cf8dcb06be_isrForCrosstalkSources_855235_33.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a5223054-fdf3-43ea-9ea4-c5cf8dcb06be_isrForCrosstalkSources_855235_33.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ca9c8b6f12fc3761322a51fa7f320af with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/04fa0369-f042-4a67-9c34-7a6e47da52c3_isrForCrosstalkSources_855233_25.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/04fa0369-f042-4a67-9c34-7a6e47da52c3_isrForCrosstalkSources_855233_25.stderr parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9606b937433b3f2617797f46c7c67f15 with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/026bf068-3a25-48d1-8632-d8ff147447ad_isrForCrosstalkSources_855156_51.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/026bf068-3a25-48d1-8632-d8ff147447ad_isrForCrosstalkSources_855156_51.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b4a51b1753b952b9aa03800393438d9 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d1a613af-2db5-49e7-8c0a-92ed09e3d617_isrForCrosstalkSources_855154_45.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d1a613af-2db5-49e7-8c0a-92ed09e3d617_isrForCrosstalkSources_855154_45.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cecf92f7b1c9c825e2cca2609d6f7121 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/9254a2af-faa5-4296-987a-a6412d646108_isrForCrosstalkSources_855236_8.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/9254a2af-faa5-4296-987a-a6412d646108_isrForCrosstalkSources_855236_8.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941def814b4ab2ece42b4178ab9074e8 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/30ca7de3-86ce-41eb-bb9c-863b635d468b_isrForCrosstalkSources_855154_39.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/30ca7de3-86ce-41eb-bb9c-863b635d468b_isrForCrosstalkSources_855154_39.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58fe9d5e74af3ccf49b5410f2d630fa4 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8a12c318-db95-4b68-9ed2-9bbe0923fa55_isrForCrosstalkSources_855238_18.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8a12c318-db95-4b68-9ed2-9bbe0923fa55_isrForCrosstalkSources_855238_18.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c87b17564c6a948f3dc5c89464a0ab with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/002b2d3a-db57-4eb1-90d4-1e706bf48864_isrForCrosstalkSources_855242_39.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/002b2d3a-db57-4eb1-90d4-1e706bf48864_isrForCrosstalkSources_855242_39.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3c5f50f23d40772072e51f9f61edf90 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2e3456a5-fd84-4a19-82e5-fdb025f39ccf_isrForCrosstalkSources_855240_19.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2e3456a5-fd84-4a19-82e5-fdb025f39ccf_isrForCrosstalkSources_855240_19.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d214ba6e9810c6d87244ec3d1e7cd0fc with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8522deea-16c2-4761-8ca3-5e8ef7a7baa4_isrForCrosstalkSources_855237_2.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/8522deea-16c2-4761-8ca3-5e8ef7a7baa4_isrForCrosstalkSources_855237_2.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 257d4e0286a036aa6bfc47d8dae8ed4c with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/88f1b536-15b7-4055-b9ca-ec87b2d9d2f3_isrForCrosstalkSources_855238_10.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/88f1b536-15b7-4055-b9ca-ec87b2d9d2f3_isrForCrosstalkSources_855238_10.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d188027c17af97a5897712a835ed087 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a9a08ce0-6d3e-4ac5-a7df-3dcc0d0397cf_isrForCrosstalkSources_855238_51.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/a9a08ce0-6d3e-4ac5-a7df-3dcc0d0397cf_isrForCrosstalkSources_855238_51.stderr parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3172fbd22299e1147ce12cae995b6739 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d8b2c286-8739-4d34-bc3f-8f16f5e6fba5_isrForCrosstalkSources_855241_36.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d8b2c286-8739-4d34-bc3f-8f16f5e6fba5_isrForCrosstalkSources_855241_36.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d384d83008cbe2fffef8062b521e6b76 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/debf286a-95a1-421a-95f1-73520f3c4d23_isrForCrosstalkSources_855231_11.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/debf286a-95a1-421a-95f1-73520f3c4d23_isrForCrosstalkSources_855231_11.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90ab47cf914c9b9e19789ce845ff7d4a with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b39de129-85cd-4775-bb52-e1742602c1a8_isrForCrosstalkSources_855154_7.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b39de129-85cd-4775-bb52-e1742602c1a8_isrForCrosstalkSources_855154_7.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0108b27952274104cb8315a05de8539 with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/2c23ea9b-e830-4c54-bdad-b95513d0faf7_isrForCrosstalkSources_855236_37.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/2c23ea9b-e830-4c54-bdad-b95513d0faf7_isrForCrosstalkSources_855236_37.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5be368222ce786cb9940791bf3d4ec6f with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/19b19c96-0d8b-4a1b-ab04-bb15141824fb_isrForCrosstalkSources_855233_23.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/19b19c96-0d8b-4a1b-ab04-bb15141824fb_isrForCrosstalkSources_855233_23.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a368548162f4f71562d8f00f961e9fe5 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/7d00d7de-cf4c-4b29-bb54-96bdfeb7d41b_isrForCrosstalkSources_855235_21.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/7d00d7de-cf4c-4b29-bb54-96bdfeb7d41b_isrForCrosstalkSources_855235_21.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38df95775682ecf3d2b004ec456fd68e with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5e3b851f-f9ee-4174-9b04-d2b042367a1c_isrForCrosstalkSources_855235_40.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/5e3b851f-f9ee-4174-9b04-d2b042367a1c_isrForCrosstalkSources_855235_40.stderr 2024-03-08 10:02:36 proc_lsst.multi:170 [INFO] [multi] job_id 16914788 proc_lsst.multi INFO: [multi] job_id 16914788 2024-03-08 10:02:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-08 10:02:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914788 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914788 parsl.executors.status_handling DEBUG: Launched block 167 on executor multi with job ID 16914788 parsl.executors.status_handling INFO: Allocated block ID 168 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-168 2024-03-08 10:02:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-168 2024-03-08 10:02:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 111 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 111 2024-03-08 10:02:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=168 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34078ba734d6cb8723b3a1aa94a88110 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/242321fb-223d-48c8-9919-3c99b4d9ef78_isrForCrosstalkSources_855241_11.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/242321fb-223d-48c8-9919-3c99b4d9ef78_isrForCrosstalkSources_855241_11.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dba55b4f8a3f05d6a6e221ae17f2a55 with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/209025bf-3dc1-43fb-8e22-c21856208635_isrForCrosstalkSources_855241_2.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/209025bf-3dc1-43fb-8e22-c21856208635_isrForCrosstalkSources_855241_2.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b16fbe052f171a4df72c8dde50eb927 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/58d51597-aa1f-4225-9002-59c61b7ca4ce_isrForCrosstalkSources_855235_2.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/58d51597-aa1f-4225-9002-59c61b7ca4ce_isrForCrosstalkSources_855235_2.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46750c56180135c4ee4602f01abdc9ca with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c7d602f7-4fa2-4022-bebb-5d61230614d2_isrForCrosstalkSources_855242_19.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c7d602f7-4fa2-4022-bebb-5d61230614d2_isrForCrosstalkSources_855242_19.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8acc02fec6bcc5aefd96991362c5f2b4 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24124ca8-5fbe-4242-b4af-37fc2ecb2414_isrForCrosstalkSources_855241_28.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/24124ca8-5fbe-4242-b4af-37fc2ecb2414_isrForCrosstalkSources_855241_28.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2d34447d103fbe092b8127fa9bc4507 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/75d8db9e-a409-42ec-a678-d4cebfb35698_isrForCrosstalkSources_855235_9.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/75d8db9e-a409-42ec-a678-d4cebfb35698_isrForCrosstalkSources_855235_9.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d000656949e869fa66781d4a74310a5 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/ffa58946-5b37-4ecd-bc66-ec8449149783_isrForCrosstalkSources_855241_62.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/ffa58946-5b37-4ecd-bc66-ec8449149783_isrForCrosstalkSources_855241_62.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05283f7180051709cbc09711486c0026 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/2f80c4cc-eae3-42c0-95f3-93c518d01776_isrForCrosstalkSources_855156_56.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/2f80c4cc-eae3-42c0-95f3-93c518d01776_isrForCrosstalkSources_855156_56.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fff39773655d620052209854e02f785e with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/25e59c92-fad4-47fe-9f75-4fd1258c7dbd_isrForCrosstalkSources_855239_16.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/25e59c92-fad4-47fe-9f75-4fd1258c7dbd_isrForCrosstalkSources_855239_16.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2b357bb8e3c921039351ee112c98f69 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a664abce-f5f1-4011-8d0e-14d829a9ed10_isrForCrosstalkSources_855237_49.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/a664abce-f5f1-4011-8d0e-14d829a9ed10_isrForCrosstalkSources_855237_49.stderr parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7093eb7d92299527bb2c96e2349710 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/78b77a16-5fe1-4031-9b87-e31913e6fc49_isrForCrosstalkSources_855155_5.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/78b77a16-5fe1-4031-9b87-e31913e6fc49_isrForCrosstalkSources_855155_5.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d3b8ce244d01487c3c18367d6272099 with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3069e4bf-e145-4211-aa83-7f959c39c54d_isrForCrosstalkSources_855155_37.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3069e4bf-e145-4211-aa83-7f959c39c54d_isrForCrosstalkSources_855155_37.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76cbfb08edaf6c112e3ce2d584abc42c with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2c6fb920-ddf2-4cd0-9fef-6ec255b16778_isrForCrosstalkSources_855234_43.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2c6fb920-ddf2-4cd0-9fef-6ec255b16778_isrForCrosstalkSources_855234_43.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 639ad9a0e99be9cbb6ce373470fb1e82 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b10944c2-4d79-47ec-b5ad-d6157992ce13_isrForCrosstalkSources_855242_43.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b10944c2-4d79-47ec-b5ad-d6157992ce13_isrForCrosstalkSources_855242_43.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dce32e16f917ae2e3870c7579c30c663 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bc22ebbb-acae-4f76-809a-eff78cfac1c5_isrForCrosstalkSources_855236_59.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/bc22ebbb-acae-4f76-809a-eff78cfac1c5_isrForCrosstalkSources_855236_59.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f2a5344240dc5d7f2c9a5c23027f600 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7781fdbe-6df3-422b-925e-5cd09443ec5b_isrForCrosstalkSources_855154_11.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/7781fdbe-6df3-422b-925e-5cd09443ec5b_isrForCrosstalkSources_855154_11.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a83f7bb242cda85b450c7625b49db2ee with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/339f9ae9-f86c-4fce-b64a-d23725fabf87_isrForCrosstalkSources_855235_48.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/339f9ae9-f86c-4fce-b64a-d23725fabf87_isrForCrosstalkSources_855235_48.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4c32b9237576b3e19333f8b8afca7cd with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0cc662d6-be6b-4243-b6d7-15a88e3a8b92_isrForCrosstalkSources_855242_58.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/0cc662d6-be6b-4243-b6d7-15a88e3a8b92_isrForCrosstalkSources_855242_58.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90885078dba5156c53e17e878aaadbdf with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/55b552b6-be26-458e-96cc-a6ac1cdef193_isrForCrosstalkSources_855231_29.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/55b552b6-be26-458e-96cc-a6ac1cdef193_isrForCrosstalkSources_855231_29.stderr 2024-03-08 10:02:37 proc_lsst.multi:170 [INFO] [multi] job_id 16914789 proc_lsst.multi INFO: [multi] job_id 16914789 2024-03-08 10:02:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-08 10:02:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914789 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914789 parsl.executors.status_handling DEBUG: Launched block 168 on executor multi with job ID 16914789 parsl.executors.status_handling INFO: Allocated block ID 169 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-169 2024-03-08 10:02:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-169 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 112 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 112 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=169 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcfda2bc8a4dc763e38d1d7ad1d6e4b4 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/703936dd-2629-4a56-a95c-7060d5725b52_isrForCrosstalkSources_855236_16.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/703936dd-2629-4a56-a95c-7060d5725b52_isrForCrosstalkSources_855236_16.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7f0c3a13186e4f303f2deb91483c101 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/fd167fcc-6b84-4dc8-bb8f-8542c9eb2757_isrForCrosstalkSources_855233_1.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/fd167fcc-6b84-4dc8-bb8f-8542c9eb2757_isrForCrosstalkSources_855233_1.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a052a1c2c7f60948c8b320acfc9f344 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/533ed82d-4247-4d23-ad72-006cc1726ca8_isrForCrosstalkSources_855233_41.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/533ed82d-4247-4d23-ad72-006cc1726ca8_isrForCrosstalkSources_855233_41.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00b903c2c3f0cb6390dfd21aea826824 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f21615e7-dd0b-4d9a-bcd6-ea5b4846d0e8_isrForCrosstalkSources_855240_3.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f21615e7-dd0b-4d9a-bcd6-ea5b4846d0e8_isrForCrosstalkSources_855240_3.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f8d27dbbe7bf76c31497a1ca3cead5a with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e7d49e2c-ca26-41d1-bceb-e7e239df0f46_isrForCrosstalkSources_855231_55.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/e7d49e2c-ca26-41d1-bceb-e7e239df0f46_isrForCrosstalkSources_855231_55.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f1f2abf2e823d52541098671a5ab69 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f3cae9a2-2414-4cda-b48f-e1e61fbe6ea7_isrForCrosstalkSources_855233_50.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/f3cae9a2-2414-4cda-b48f-e1e61fbe6ea7_isrForCrosstalkSources_855233_50.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f55a03cb3ed6aa74b65528f6456f640 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/55961318-52a0-483f-bb4f-5f5e9e246374_isrForCrosstalkSources_855241_59.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/55961318-52a0-483f-bb4f-5f5e9e246374_isrForCrosstalkSources_855241_59.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac60ecd8be89090c423aad6a13c51c1c with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a075b46-38e2-47f7-bae7-64123ec6d60d_isrForCrosstalkSources_855237_38.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9a075b46-38e2-47f7-bae7-64123ec6d60d_isrForCrosstalkSources_855237_38.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c37a0c3cdddd278a5e81e069867fab3d with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a5bb0e98-453d-4da5-b10b-dd4c6df91125_isrForCrosstalkSources_855233_2.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a5bb0e98-453d-4da5-b10b-dd4c6df91125_isrForCrosstalkSources_855233_2.stderr 2024-03-08 10:02:37 proc_lsst.multi:170 [INFO] [multi] job_id 16914790 proc_lsst.multi INFO: [multi] job_id 16914790 2024-03-08 10:02:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-08 10:02:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914790 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914790 parsl.executors.status_handling DEBUG: Launched block 169 on executor multi with job ID 16914790 parsl.executors.status_handling INFO: Allocated block ID 170 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-170 2024-03-08 10:02:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-170 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 113 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 113 2024-03-08 10:02:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=170 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e3ae630e94b68ca9ee7170cd3fbe938 with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/480756e3-dbb9-49f2-b651-924067d52eb4_isrForCrosstalkSources_855155_13.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/480756e3-dbb9-49f2-b651-924067d52eb4_isrForCrosstalkSources_855155_13.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31efbf0e41db70092ae073e7cd01b6a8 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0811f197-6794-47fc-82d7-be25d980dec1_isrForCrosstalkSources_855241_48.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/0811f197-6794-47fc-82d7-be25d980dec1_isrForCrosstalkSources_855241_48.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b531d11fd68aa5749d21eb5880fc87ed with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/4915eef4-b636-4b50-946a-2e5fd23e23f5_isrForCrosstalkSources_855239_12.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/4915eef4-b636-4b50-946a-2e5fd23e23f5_isrForCrosstalkSources_855239_12.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68e4023ddcae45206f9af5a07d556f41 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f9c2d9b4-9d24-4d33-8777-a6ecfb1464c7_isrForCrosstalkSources_855242_23.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/f9c2d9b4-9d24-4d33-8777-a6ecfb1464c7_isrForCrosstalkSources_855242_23.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6ca901c944ebcff26fe6d4a234021dc with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7b79bb10-cf4f-4662-bc33-0d6358504b46_isrForCrosstalkSources_855241_60.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7b79bb10-cf4f-4662-bc33-0d6358504b46_isrForCrosstalkSources_855241_60.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf90fb73cf5cae471f9db134f40da217 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b0eb0223-d739-48a2-adad-933ea204c962_isrForCrosstalkSources_855242_44.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/b0eb0223-d739-48a2-adad-933ea204c962_isrForCrosstalkSources_855242_44.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76aa643a7be46996ac95820b91648f11 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/bbacf5ae-b941-4954-9068-a0a74d3a33ac_isrForCrosstalkSources_855235_50.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/bbacf5ae-b941-4954-9068-a0a74d3a33ac_isrForCrosstalkSources_855235_50.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77f09767fe42d5681e42a7cc2d00afd8 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9f8cf1f2-bbd5-41b2-aa87-5c6beab8e7d4_isrForCrosstalkSources_855231_25.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9f8cf1f2-bbd5-41b2-aa87-5c6beab8e7d4_isrForCrosstalkSources_855231_25.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 230f4bb3ec3e1b3244299b2352117a1d with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/da9a0767-d346-4cf4-bb57-7dc5712045d6_isrForCrosstalkSources_855156_25.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/da9a0767-d346-4cf4-bb57-7dc5712045d6_isrForCrosstalkSources_855156_25.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fff9101b0e74aeba4e50c8440ad2cdc with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/22261aab-5937-4545-8a82-d895c98ccbac_isrForCrosstalkSources_855240_52.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/22261aab-5937-4545-8a82-d895c98ccbac_isrForCrosstalkSources_855240_52.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 542a7dfc636f5bd9a48e27de6adf716e with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0561a62f-9cf9-409f-8dd3-26fb6d9ba15e_isrForCrosstalkSources_855239_56.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0561a62f-9cf9-409f-8dd3-26fb6d9ba15e_isrForCrosstalkSources_855239_56.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adcb682da7c11db8baebe91455f4e844 with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a6e79388-9251-4245-9fde-106009e3d415_isrForCrosstalkSources_855239_60.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a6e79388-9251-4245-9fde-106009e3d415_isrForCrosstalkSources_855239_60.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 630f5d4661ac5c6616b996fcfbff2e05 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d0399232-073b-40ad-a8c1-937ec1e59082_isrForCrosstalkSources_855233_9.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d0399232-073b-40ad-a8c1-937ec1e59082_isrForCrosstalkSources_855233_9.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf7844de34d2ba1477677434a7ae767 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/83c5ebd4-ec73-4be1-b4ba-b02a36b03f08_isrForCrosstalkSources_855239_7.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/83c5ebd4-ec73-4be1-b4ba-b02a36b03f08_isrForCrosstalkSources_855239_7.stderr parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de9c6795ce5b249e328d49eed631c121 with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4409821b-c831-48fe-891e-0d926c3756b3_isrForCrosstalkSources_855237_27.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4409821b-c831-48fe-891e-0d926c3756b3_isrForCrosstalkSources_855237_27.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c763ff80568f6b374595eda63703cc83 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d9b451de-d04c-4738-88a1-28d187057b91_isrForCrosstalkSources_855236_51.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d9b451de-d04c-4738-88a1-28d187057b91_isrForCrosstalkSources_855236_51.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53b8ac76e3a50e0a77aa8c0539b6f6bd with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c77f7b27-8225-45bf-a3e2-9ff49e240fb2_isrForCrosstalkSources_855237_47.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c77f7b27-8225-45bf-a3e2-9ff49e240fb2_isrForCrosstalkSources_855237_47.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07ae3a43fe21a1cf6c553f8d75e15c21 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a1837d3f-0886-40bc-a9c6-2d693d4be828_isrForCrosstalkSources_855155_42.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/a1837d3f-0886-40bc-a9c6-2d693d4be828_isrForCrosstalkSources_855155_42.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0617403645f429564036a75d5d8d0eb4 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2cde778-28a8-4421-b59c-cdc0ab860c5c_isrForCrosstalkSources_855155_26.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/b2cde778-28a8-4421-b59c-cdc0ab860c5c_isrForCrosstalkSources_855155_26.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53f5b32a13c64352fe6c5dc7cb49bddf with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/88e013fe-ff19-4761-bb8d-ae1334ec4329_isrForCrosstalkSources_855234_18.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/88e013fe-ff19-4761-bb8d-ae1334ec4329_isrForCrosstalkSources_855234_18.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90e2f6a56f2ab13ab98e868fa4dc7515 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f89a2aa2-5632-4da6-9de6-c9ba4b098148_isrForCrosstalkSources_855234_13.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/f89a2aa2-5632-4da6-9de6-c9ba4b098148_isrForCrosstalkSources_855234_13.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef0c1f8c6068250558b2fd861e587a16 with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3a49c34f-528c-4535-a362-e41ecdae145f_isrForCrosstalkSources_855234_62.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/3a49c34f-528c-4535-a362-e41ecdae145f_isrForCrosstalkSources_855234_62.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02614a4049ce14414cdcd7d4617e36f7 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d2c92434-4d94-440f-b530-1c6ad5595df8_isrForCrosstalkSources_855156_19.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d2c92434-4d94-440f-b530-1c6ad5595df8_isrForCrosstalkSources_855156_19.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f0f275529d9897f8b41b0ea93466bf4 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e94ec665-0247-4168-8626-ab2317f90a26_isrForCrosstalkSources_855237_25.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e94ec665-0247-4168-8626-ab2317f90a26_isrForCrosstalkSources_855237_25.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b93166b575ebb52e170f6b50da095f with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4eeaa554-5e30-4f02-ae83-611ba8c0462f_isrForCrosstalkSources_855236_12.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4eeaa554-5e30-4f02-ae83-611ba8c0462f_isrForCrosstalkSources_855236_12.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fc813555557a36f40aacbcba7298ba3 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ec47e380-0de0-40f6-afa3-b2d409476dd5_isrForCrosstalkSources_855234_39.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ec47e380-0de0-40f6-afa3-b2d409476dd5_isrForCrosstalkSources_855234_39.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98f5b9328c772df5e452b21ed6bd927e with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/29d0666a-78c3-4559-ac88-cd27e80cb359_isrForCrosstalkSources_855234_44.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/29d0666a-78c3-4559-ac88-cd27e80cb359_isrForCrosstalkSources_855234_44.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aef2dbfa4642218f657560b41416b201 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6d909492-d87d-4c21-98cc-32a0c311ad30_isrForCrosstalkSources_855240_22.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6d909492-d87d-4c21-98cc-32a0c311ad30_isrForCrosstalkSources_855240_22.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d5c07dc463b85fc6c567ce4315ae050 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ae0d54e2-e03e-486a-8dc8-228f8b8a4b4a_isrForCrosstalkSources_855238_8.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/ae0d54e2-e03e-486a-8dc8-228f8b8a4b4a_isrForCrosstalkSources_855238_8.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85cbd754b6296099cf19b12896f33316 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dac00e95-6222-4ec9-bad9-5962ca5c585b_isrForCrosstalkSources_855154_40.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/dac00e95-6222-4ec9-bad9-5962ca5c585b_isrForCrosstalkSources_855154_40.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bb1fa39905323173f086340fff34b2e with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1248e1e8-e028-4890-b591-e3f8a64581b1_isrForCrosstalkSources_855240_2.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1248e1e8-e028-4890-b591-e3f8a64581b1_isrForCrosstalkSources_855240_2.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 206f1a68aa65cffb741c54afa6fdf7a0 with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/442ede9c-5a4f-4657-b899-3e61cbd833b9_isrForCrosstalkSources_855236_25.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/442ede9c-5a4f-4657-b899-3e61cbd833b9_isrForCrosstalkSources_855236_25.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b1ccaaef2639af44ddbec513f2ef280 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/53f5699c-7228-47a0-a8bd-6765efa55150_isrForCrosstalkSources_855241_34.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/53f5699c-7228-47a0-a8bd-6765efa55150_isrForCrosstalkSources_855241_34.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c569e7af173a8d502ffa5c0aebb98ff with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5f56c7a5-b08d-43ad-9f15-85fad645a032_isrForCrosstalkSources_855155_44.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5f56c7a5-b08d-43ad-9f15-85fad645a032_isrForCrosstalkSources_855155_44.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68414eb2b170c9597a9a1b39ad380e7f with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1d5aed23-69c8-4662-ad57-17605999ab3c_isrForCrosstalkSources_855232_42.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/1d5aed23-69c8-4662-ad57-17605999ab3c_isrForCrosstalkSources_855232_42.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6849d34aec00b2f44180ffc0dc9fe781 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3dfd6f6f-190c-4d50-bc1b-2599320a22fb_isrForCrosstalkSources_855155_38.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3dfd6f6f-190c-4d50-bc1b-2599320a22fb_isrForCrosstalkSources_855155_38.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9094a7c85c835dcb752d781a7efb07a0 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/35cc9212-e2d4-472f-9391-c76151e3f4ff_isrForCrosstalkSources_855154_21.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/35cc9212-e2d4-472f-9391-c76151e3f4ff_isrForCrosstalkSources_855154_21.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 946860518f2ba04d91978aacb1a6fce5 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/042ac56e-84bb-4008-88c0-182c6814e5bb_isrForCrosstalkSources_855235_37.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/042ac56e-84bb-4008-88c0-182c6814e5bb_isrForCrosstalkSources_855235_37.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec5f4c118a275c1111a915b4570b7370 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5143f579-31c5-47de-9a1c-eeb851a9c6bf_isrForCrosstalkSources_855234_35.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/5143f579-31c5-47de-9a1c-eeb851a9c6bf_isrForCrosstalkSources_855234_35.stderr parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20812647b64ffbe72bfb85c018c37e2a with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a5a00542-860a-4946-9f2f-570f2b4ebb8e_isrForCrosstalkSources_855239_42.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a5a00542-860a-4946-9f2f-570f2b4ebb8e_isrForCrosstalkSources_855239_42.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43870fca62c559f7698378642e119344 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/da877c33-c23e-4ccd-8fb6-c95d2b025eed_isrForCrosstalkSources_855231_44.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/da877c33-c23e-4ccd-8fb6-c95d2b025eed_isrForCrosstalkSources_855231_44.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86927273d01dc7f49b37965f4424bd88 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/4712b25d-5bfe-427d-b5d8-60486207ca84_isrForCrosstalkSources_855155_20.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/4712b25d-5bfe-427d-b5d8-60486207ca84_isrForCrosstalkSources_855155_20.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867d5aad68006382bd1d8300adc919c7 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/81447dcb-6b6f-49f3-847c-d0cf2a842720_isrForCrosstalkSources_855233_21.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/81447dcb-6b6f-49f3-847c-d0cf2a842720_isrForCrosstalkSources_855233_21.stderr parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7593fdc2341b0dc5ed480ce6ade0f200 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2077b9f7-9fdc-4ec0-9162-6e7a1248d901_isrForCrosstalkSources_855154_10.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2077b9f7-9fdc-4ec0-9162-6e7a1248d901_isrForCrosstalkSources_855154_10.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0083eeda0631556c69d0d9695e6ee738 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/434b97ea-099e-459a-976b-864ad9a9e0b2_isrForCrosstalkSources_855156_61.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/434b97ea-099e-459a-976b-864ad9a9e0b2_isrForCrosstalkSources_855156_61.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d96257717caa3a3e186896dfbf93c831 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5b5f178e-ac1e-4044-b035-bf811b6fb3cb_isrForCrosstalkSources_855155_35.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/5b5f178e-ac1e-4044-b035-bf811b6fb3cb_isrForCrosstalkSources_855155_35.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e86d59426512634ef82f908d06cc6ac with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6b57054f-9fa3-439d-b03b-11b557b7895c_isrForCrosstalkSources_855240_1.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/6b57054f-9fa3-439d-b03b-11b557b7895c_isrForCrosstalkSources_855240_1.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bf18187fcd368989505dad78e2d544b with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/dc6ce576-cc94-4fdf-bb12-855377810786_isrForCrosstalkSources_855232_45.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/dc6ce576-cc94-4fdf-bb12-855377810786_isrForCrosstalkSources_855232_45.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29d95e02c8c2cd5b8eed1d9e76b03cb3 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a6c085c6-41b7-4fbc-8088-4d90735e5814_isrForCrosstalkSources_855242_60.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/a6c085c6-41b7-4fbc-8088-4d90735e5814_isrForCrosstalkSources_855242_60.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03eaf105d26f9782785c62b4b43822d8 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/fab72498-2c27-44c3-a540-32d7e7a2877f_isrForCrosstalkSources_855156_54.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/fab72498-2c27-44c3-a540-32d7e7a2877f_isrForCrosstalkSources_855156_54.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c363a2a8fff74d5a5f3d06ee8619224 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/068f5736-edb2-4a50-bcea-19ef18a08c4a_isrForCrosstalkSources_855241_26.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/068f5736-edb2-4a50-bcea-19ef18a08c4a_isrForCrosstalkSources_855241_26.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f57eff13e5a3197691d556a00af31382 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/494cd584-9775-46ff-a685-a43e68fb85bc_isrForCrosstalkSources_855236_48.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/494cd584-9775-46ff-a685-a43e68fb85bc_isrForCrosstalkSources_855236_48.stderr parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeb1c73420d8d01ea57c1919aee11310 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b369ea40-cc33-4d0c-91b1-693363ec2ded_isrForCrosstalkSources_855233_19.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b369ea40-cc33-4d0c-91b1-693363ec2ded_isrForCrosstalkSources_855233_19.stderr parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69a5c31f8e25cce6927b1dee44f4c206 with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/57622a83-55f7-43a7-8e2a-ff304e4291aa_isrForCrosstalkSources_855234_56.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/57622a83-55f7-43a7-8e2a-ff304e4291aa_isrForCrosstalkSources_855234_56.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e74ee1a3d2b4b61b74a49a3114f9b443 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/60639732-21a2-4e21-91b0-8a7274826537_isrForCrosstalkSources_855239_9.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/60639732-21a2-4e21-91b0-8a7274826537_isrForCrosstalkSources_855239_9.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7b6e75bd4550d425748c6c440183077 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/379a8651-8fcb-49a9-9d94-99fa4a24d115_isrForCrosstalkSources_855231_34.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/379a8651-8fcb-49a9-9d94-99fa4a24d115_isrForCrosstalkSources_855231_34.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8efe90bef540b07a3dcfcffc9f342a73 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a00e7ff9-7c9c-4918-82ec-e84a1116fd87_isrForCrosstalkSources_855156_38.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a00e7ff9-7c9c-4918-82ec-e84a1116fd87_isrForCrosstalkSources_855156_38.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd2abca384fbe02d4c816a796ed08425 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/24aa56f1-a5e9-438e-a679-b3811e075228_isrForCrosstalkSources_855156_47.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/24aa56f1-a5e9-438e-a679-b3811e075228_isrForCrosstalkSources_855156_47.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed34a1307fba576617787ad661196c9c with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2366332e-dda8-4328-a2e8-9af2931a1530_isrForCrosstalkSources_855242_15.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/2366332e-dda8-4328-a2e8-9af2931a1530_isrForCrosstalkSources_855242_15.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1b476e7bc314cc36b9fd65704fe4d07 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/42c24ab8-7224-45d5-af73-3f1630801fa3_isrForCrosstalkSources_855154_57.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/42c24ab8-7224-45d5-af73-3f1630801fa3_isrForCrosstalkSources_855154_57.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f7d8cdae87c4f84bfc848df15a98083 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1486ffbd-7fb0-4182-896c-ec3e256ec271_isrForCrosstalkSources_855237_18.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1486ffbd-7fb0-4182-896c-ec3e256ec271_isrForCrosstalkSources_855237_18.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15926d1e1b95901d985c37ecaff2e8d0 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3aed75e2-963a-4d22-9097-56e462db2c61_isrForCrosstalkSources_855231_54.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/3aed75e2-963a-4d22-9097-56e462db2c61_isrForCrosstalkSources_855231_54.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f46b3532940e206d460fc04c1bb8e167 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e78992a9-1501-4e1b-abdc-b5f55474758c_isrForCrosstalkSources_855232_38.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/e78992a9-1501-4e1b-abdc-b5f55474758c_isrForCrosstalkSources_855232_38.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d97087df8221872a166c2371f13dc3 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/22a24431-1174-47cd-a24c-bade86d0c01c_isrForCrosstalkSources_855242_16.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/22a24431-1174-47cd-a24c-bade86d0c01c_isrForCrosstalkSources_855242_16.stderr parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 166a1d6ad0825bd74b648d0c393afc12 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d9b35254-3e51-4f5a-ad9d-ec8c6d26bb62_isrForCrosstalkSources_855232_29.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d9b35254-3e51-4f5a-ad9d-ec8c6d26bb62_isrForCrosstalkSources_855232_29.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0459c5fe9f401a1046952ed172e1dab2 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7c4aceb8-ec33-462c-989c-2d57c4b3b515_isrForCrosstalkSources_855232_28.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7c4aceb8-ec33-462c-989c-2d57c4b3b515_isrForCrosstalkSources_855232_28.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ebb4c8a06bf91d9873584756e884c80 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/12d3d6a9-1101-4886-935f-7b376a5bda7c_isrForCrosstalkSources_855156_57.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/12d3d6a9-1101-4886-935f-7b376a5bda7c_isrForCrosstalkSources_855156_57.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c771ee8ec84ce749e3b6b62642fbdf05 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/461d0cf0-f204-46c8-9280-e2bf59f61002_isrForCrosstalkSources_855238_45.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/461d0cf0-f204-46c8-9280-e2bf59f61002_isrForCrosstalkSources_855238_45.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c7e584adb579e7ff2097938d37f5321 with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/38e40d28-5d4d-4c38-8c9e-e05077677cce_isrForCrosstalkSources_855233_33.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/38e40d28-5d4d-4c38-8c9e-e05077677cce_isrForCrosstalkSources_855233_33.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dc60906b8adc51c11dd14c3be19d264 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/192b3f35-f64f-46f2-b809-f00572758f12_isrForCrosstalkSources_855156_12.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/192b3f35-f64f-46f2-b809-f00572758f12_isrForCrosstalkSources_855156_12.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec4cf43fa753d8a0efdb47437827e111 with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/02abc455-ffcf-433f-9dda-4fd758f73cf2_isrForCrosstalkSources_855239_39.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/02abc455-ffcf-433f-9dda-4fd758f73cf2_isrForCrosstalkSources_855239_39.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35d05977613031e14dda27c635def449 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/93a87c44-218b-4fd3-858b-224955072785_isrForCrosstalkSources_855240_34.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/93a87c44-218b-4fd3-858b-224955072785_isrForCrosstalkSources_855240_34.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a1c996f1a8d79dd5acaf2b1444fb231 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/03855faf-086d-439e-b680-8424d72f8a62_isrForCrosstalkSources_855233_31.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/03855faf-086d-439e-b680-8424d72f8a62_isrForCrosstalkSources_855233_31.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d4d63db105aa73602f776385704c4d6 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6332ded2-2d91-436a-bde8-b451b7d962f5_isrForCrosstalkSources_855154_53.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/6332ded2-2d91-436a-bde8-b451b7d962f5_isrForCrosstalkSources_855154_53.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1e52148b84e799818f0d0df378be3da with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8ce7b13f-2df9-4477-9acb-d38eab88ba83_isrForCrosstalkSources_855236_6.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/8ce7b13f-2df9-4477-9acb-d38eab88ba83_isrForCrosstalkSources_855236_6.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66876d69aae3615405a76bf76d1bd58b with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7140d445-cc90-43eb-955f-443f214eef5a_isrForCrosstalkSources_855237_53.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/7140d445-cc90-43eb-955f-443f214eef5a_isrForCrosstalkSources_855237_53.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01ceecde77030f90f9b367bdfecfdfd2 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f7f5c047-b4e1-4d98-b40a-9f3880135964_isrForCrosstalkSources_855239_48.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/f7f5c047-b4e1-4d98-b40a-9f3880135964_isrForCrosstalkSources_855239_48.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ab0548c629cfc192c86277c01bfb67 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/c965bd3a-e8bf-44d4-8c57-4ba34e111f69_isrForCrosstalkSources_855233_10.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/c965bd3a-e8bf-44d4-8c57-4ba34e111f69_isrForCrosstalkSources_855233_10.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a40f138286f06e1b115149f81264e0ee with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ed777137-f810-42e5-af84-3fe8944cb7a7_isrForCrosstalkSources_855234_23.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/ed777137-f810-42e5-af84-3fe8944cb7a7_isrForCrosstalkSources_855234_23.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f64423b9c8b686c8c2ee44ed0863c1d with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/44304a09-107b-4ab6-ae25-dd26bbbf392f_isrForCrosstalkSources_855232_40.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/44304a09-107b-4ab6-ae25-dd26bbbf392f_isrForCrosstalkSources_855232_40.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 813c496c5f17ed30cbb53056c287c941 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/981a4276-8423-43f3-a860-18b67565df26_isrForCrosstalkSources_855232_5.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/981a4276-8423-43f3-a860-18b67565df26_isrForCrosstalkSources_855232_5.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf5e420494e3b2be5b2c30d80ee7f160 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/21d2cf13-f20a-4496-8819-7f10e32b0a65_isrForCrosstalkSources_855238_39.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/21d2cf13-f20a-4496-8819-7f10e32b0a65_isrForCrosstalkSources_855238_39.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 009ac13a0b7386d9f57f1e67706c7883 with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5e01925f-0ffb-4a9b-8139-f6937cb961f7_isrForCrosstalkSources_855156_13.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5e01925f-0ffb-4a9b-8139-f6937cb961f7_isrForCrosstalkSources_855156_13.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc9fac9c099be30c3cd01253d350f185 with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1d3d9fe2-5d8d-486d-81c6-de31d15e938d_isrForCrosstalkSources_855239_8.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1d3d9fe2-5d8d-486d-81c6-de31d15e938d_isrForCrosstalkSources_855239_8.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 197626dcf4a635fd87f94068eb0af441 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d1c63e6a-2c11-4aa1-8f4c-46dc351da93b_isrForCrosstalkSources_855236_5.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/d1c63e6a-2c11-4aa1-8f4c-46dc351da93b_isrForCrosstalkSources_855236_5.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d9f53159d59ffd2d0ec79c16f200a82 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/499516d9-bbb0-4d90-b27c-996bb24eb277_isrForCrosstalkSources_855231_8.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/499516d9-bbb0-4d90-b27c-996bb24eb277_isrForCrosstalkSources_855231_8.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ce68ca3e3e70c2381b5b3bd263693f5 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5f9fd587-8b3e-468d-b7b2-ed5d04d439bc_isrForCrosstalkSources_855156_62.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5f9fd587-8b3e-468d-b7b2-ed5d04d439bc_isrForCrosstalkSources_855156_62.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19e4581d9c0180b42dda3a4639b849e with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4f73749d-ad1f-45e3-8191-b034a6b72727_isrForCrosstalkSources_855156_48.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/4f73749d-ad1f-45e3-8191-b034a6b72727_isrForCrosstalkSources_855156_48.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feca5965e840da13219593a369921f9c with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9bb57b29-e499-40d4-999d-513bd9b3ef49_isrForCrosstalkSources_855154_31.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/9bb57b29-e499-40d4-999d-513bd9b3ef49_isrForCrosstalkSources_855154_31.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed8c0208e4e7f5103199e86c1191e8b1 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab7cd732-6448-45de-954c-a9db4fd17f57_isrForCrosstalkSources_855235_28.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ab7cd732-6448-45de-954c-a9db4fd17f57_isrForCrosstalkSources_855235_28.stderr parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dbd2e557967743619a453c44c697789 with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/77be2220-660c-42e3-9e98-580fac7c4f4d_isrForCrosstalkSources_855234_42.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/77be2220-660c-42e3-9e98-580fac7c4f4d_isrForCrosstalkSources_855234_42.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09b6d0a828859e28d25e3d3b2f5083c7 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4d9edb75-7373-44ff-9020-81da6233d6c7_isrForCrosstalkSources_855242_42.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4d9edb75-7373-44ff-9020-81da6233d6c7_isrForCrosstalkSources_855242_42.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2db9b9a1df8093b67a3027fc6bfc51b with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aab6d2b4-adf8-4818-83df-f8530fba4df6_isrForCrosstalkSources_855242_28.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/aab6d2b4-adf8-4818-83df-f8530fba4df6_isrForCrosstalkSources_855242_28.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa4d8b7cf0ab44d130c5043f2fc6ba84 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5f8751f4-7f3b-4132-bac7-713cb54eed40_isrForCrosstalkSources_855242_54.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/5f8751f4-7f3b-4132-bac7-713cb54eed40_isrForCrosstalkSources_855242_54.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59001f472c48f71c5ca20af96535baed with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/3ed318ba-ff26-48b7-8937-e45cc87be076_isrForCrosstalkSources_855238_29.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/3ed318ba-ff26-48b7-8937-e45cc87be076_isrForCrosstalkSources_855238_29.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cbb4b7d8f0bd3a5f2cdd7327809983c with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/4b6650c3-ded7-4fca-9656-31576ef427d7_isrForCrosstalkSources_855235_54.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/4b6650c3-ded7-4fca-9656-31576ef427d7_isrForCrosstalkSources_855235_54.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e60839c390c6dde77ac386361c67cda with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e72ccc27-a9f8-4267-bae1-c2772aef2ba5_isrForCrosstalkSources_855235_46.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/e72ccc27-a9f8-4267-bae1-c2772aef2ba5_isrForCrosstalkSources_855235_46.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b671d6aa66e6acbcd88b006e079d63d3 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/def94df1-5dfe-44f5-9715-b2030e7b9e67_isrForCrosstalkSources_855234_47.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/def94df1-5dfe-44f5-9715-b2030e7b9e67_isrForCrosstalkSources_855234_47.stderr parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2609c6fa95cd47dda45d8455eb26656a with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d89d3e97-5202-438b-bbe9-e6628fff7a6b_isrForCrosstalkSources_855234_50.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d89d3e97-5202-438b-bbe9-e6628fff7a6b_isrForCrosstalkSources_855234_50.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a1eb4d7536abdfa282af056db0f6048 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b2111997-2428-402a-a7e7-da518fc904f8_isrForCrosstalkSources_855154_44.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/b2111997-2428-402a-a7e7-da518fc904f8_isrForCrosstalkSources_855154_44.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2400fe93e3e5ca2c87d79e8646c300ad with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8a79515b-6e1b-4317-9204-5611b578b618_isrForCrosstalkSources_855154_35.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/8a79515b-6e1b-4317-9204-5611b578b618_isrForCrosstalkSources_855154_35.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfafd16f44ce426d97c8bad7356a8156 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/5cd183af-8a9c-4817-9784-601c6a73e62d_isrForCrosstalkSources_855241_58.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/5cd183af-8a9c-4817-9784-601c6a73e62d_isrForCrosstalkSources_855241_58.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0efa58a4a15864bff1bebe7b3559c9ab with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/80a5f7b4-b449-43e2-92c1-4764acde3785_isrForCrosstalkSources_855233_40.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/80a5f7b4-b449-43e2-92c1-4764acde3785_isrForCrosstalkSources_855233_40.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 309d95b2e383db1667622c8d9b3c63b6 with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f34faa7a-2202-4293-84fb-91cbc43a585a_isrForCrosstalkSources_855232_13.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f34faa7a-2202-4293-84fb-91cbc43a585a_isrForCrosstalkSources_855232_13.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 200e6196c92dafd33f10881ed4691df0 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2f217c32-21a2-49a3-96dd-a09c9ec34819_isrForCrosstalkSources_855154_9.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/2f217c32-21a2-49a3-96dd-a09c9ec34819_isrForCrosstalkSources_855154_9.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0966b4ab32da0803a6b2b3a0661a8d with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ca793160-79d7-4be2-a5e8-723bdf77adea_isrForCrosstalkSources_855240_33.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/ca793160-79d7-4be2-a5e8-723bdf77adea_isrForCrosstalkSources_855240_33.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af3c5519d30912270bba09e69bfa4f47 with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e29b3d28-f0dd-47a1-b0a5-c596402b1199_isrForCrosstalkSources_855242_22.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/e29b3d28-f0dd-47a1-b0a5-c596402b1199_isrForCrosstalkSources_855242_22.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f0ff11d893c13fc236244e386ffe1f6 with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a545c06f-16e3-4756-8385-23e99e9be439_isrForCrosstalkSources_855233_34.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/a545c06f-16e3-4756-8385-23e99e9be439_isrForCrosstalkSources_855233_34.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 973bfc853a697255f4382b6752a3d62c with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0ea5beef-5579-405c-b01b-29471a2f8bfb_isrForCrosstalkSources_855239_18.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/0ea5beef-5579-405c-b01b-29471a2f8bfb_isrForCrosstalkSources_855239_18.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d05c7c484d41e3e933c3ccacb74bbf87 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d89a56a5-a814-4069-adc9-21f46c1ec1a4_isrForCrosstalkSources_855232_16.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/d89a56a5-a814-4069-adc9-21f46c1ec1a4_isrForCrosstalkSources_855232_16.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a36369c3c189058ce2015ed413fb57de with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4ecbbc3c-5f27-412b-93eb-569c40941bc0_isrForCrosstalkSources_855237_22.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/4ecbbc3c-5f27-412b-93eb-569c40941bc0_isrForCrosstalkSources_855237_22.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ec4a36d5663cb631b4ae494bb7eb604 with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/829c9f74-2ecf-43f3-b12a-5ef50bc51f26_isrForCrosstalkSources_855236_40.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/829c9f74-2ecf-43f3-b12a-5ef50bc51f26_isrForCrosstalkSources_855236_40.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 250544eb245391ebe3d01bb4039ae4ec with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6e1fa75b-4635-40a2-aa52-a756d4e461af_isrForCrosstalkSources_855235_19.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6e1fa75b-4635-40a2-aa52-a756d4e461af_isrForCrosstalkSources_855235_19.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 671975d346a45ca0e0574976921afa83 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/43cba040-6952-44b5-accb-d6adc1abe8ed_isrForCrosstalkSources_855231_18.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/43cba040-6952-44b5-accb-d6adc1abe8ed_isrForCrosstalkSources_855231_18.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c971017c0af38a2f6c6254f180a6770d with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/caaed011-6586-4d1e-b07c-1d7162994416_isrForCrosstalkSources_855154_27.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/caaed011-6586-4d1e-b07c-1d7162994416_isrForCrosstalkSources_855154_27.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9205c47af9dd0307e1ec7e414d86237 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/59353850-319f-4614-ab6e-f342406a5e6e_isrForCrosstalkSources_855238_58.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/59353850-319f-4614-ab6e-f342406a5e6e_isrForCrosstalkSources_855238_58.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aeafbc77246e9b0b915b611f3ab7808 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/72b0e870-4e18-42eb-a82a-6aa2552e35b5_isrForCrosstalkSources_855242_24.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/72b0e870-4e18-42eb-a82a-6aa2552e35b5_isrForCrosstalkSources_855242_24.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07cf7a164ee23eb279f31b58ace425b9 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d8d25cfe-1898-4218-90f9-f410bf825623_isrForCrosstalkSources_855238_19.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d8d25cfe-1898-4218-90f9-f410bf825623_isrForCrosstalkSources_855238_19.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 236e0a02a1c55215d6729000a2e43d58 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/668ef56d-c868-4ded-857d-d227669b071f_isrForCrosstalkSources_855239_54.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/668ef56d-c868-4ded-857d-d227669b071f_isrForCrosstalkSources_855239_54.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb77baf69141525386febc1940fe5a74 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/dadadd3c-a810-414c-830e-7778e9a8bbf2_isrForCrosstalkSources_855239_53.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/dadadd3c-a810-414c-830e-7778e9a8bbf2_isrForCrosstalkSources_855239_53.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9386abb98f61766a790e88ed5cfbc9a with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/31f3de11-5fc0-4746-9754-d11d32bcb2f7_isrForCrosstalkSources_855236_27.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/31f3de11-5fc0-4746-9754-d11d32bcb2f7_isrForCrosstalkSources_855236_27.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bee9f4389a1a5f8342196fbc562b7fd2 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f919ea81-1511-4c87-b554-7dee80903211_isrForCrosstalkSources_855232_54.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f919ea81-1511-4c87-b554-7dee80903211_isrForCrosstalkSources_855232_54.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 846a0da31fef11404646ebcdd4d43a7a with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0d31442e-09d1-4747-ade7-21d3ece07c70_isrForCrosstalkSources_855231_40.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/0d31442e-09d1-4747-ade7-21d3ece07c70_isrForCrosstalkSources_855231_40.stderr 2024-03-08 10:02:42 proc_lsst.multi:170 [INFO] [multi] job_id 16914791 proc_lsst.multi INFO: [multi] job_id 16914791 2024-03-08 10:02:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-08 10:02:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914791 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914791 parsl.executors.status_handling DEBUG: Launched block 170 on executor multi with job ID 16914791 parsl.executors.status_handling INFO: Allocated block ID 171 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-171 2024-03-08 10:02:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-171 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 114 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 114 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=171 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a59db2ddf527f222b1c31367d114ffe with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8446c04d-97d6-430b-904b-903ff9fb84fd_isrForCrosstalkSources_855238_11.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/8446c04d-97d6-430b-904b-903ff9fb84fd_isrForCrosstalkSources_855238_11.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4bc95a81c938565e599206342839d03 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1dc92e6d-0db8-4e87-b6aa-2a6b180402b9_isrForCrosstalkSources_855237_55.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1dc92e6d-0db8-4e87-b6aa-2a6b180402b9_isrForCrosstalkSources_855237_55.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70d7caa53a8f2bdaf38f6bbf67bc44b4 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/16e17670-6667-49d0-b281-6bd2e9619054_isrForCrosstalkSources_855237_6.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/16e17670-6667-49d0-b281-6bd2e9619054_isrForCrosstalkSources_855237_6.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d1bf5439112011d7fcf7020d7507759 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bff693f5-7620-4b3b-9f28-752d44c46b4a_isrForCrosstalkSources_855237_14.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bff693f5-7620-4b3b-9f28-752d44c46b4a_isrForCrosstalkSources_855237_14.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf65aef0ebd68bdb7369aac1d4b4eeda with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f791457d-5ab4-46f5-a2b8-7af34ef4a56c_isrForCrosstalkSources_855240_4.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/f791457d-5ab4-46f5-a2b8-7af34ef4a56c_isrForCrosstalkSources_855240_4.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c1300eab5445cc264153d2c04d17944 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a303ec76-bc78-455e-a1dd-d91a3628e210_isrForCrosstalkSources_855234_45.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a303ec76-bc78-455e-a1dd-d91a3628e210_isrForCrosstalkSources_855234_45.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a486878565b376798a0362718981924b with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5f2c6fcc-5224-457d-9c4f-04097512d12e_isrForCrosstalkSources_855236_13.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/5f2c6fcc-5224-457d-9c4f-04097512d12e_isrForCrosstalkSources_855236_13.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20cc64802447aa72881e9f7143b4761e with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2fdd689a-e4f8-40ad-a2de-23461ca1b99c_isrForCrosstalkSources_855239_26.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2fdd689a-e4f8-40ad-a2de-23461ca1b99c_isrForCrosstalkSources_855239_26.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daf5a4c345dbf0f76989ea4a84398744 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/93aa32b2-f04b-4bd5-8863-410fd5e2e80d_isrForCrosstalkSources_855236_9.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/93aa32b2-f04b-4bd5-8863-410fd5e2e80d_isrForCrosstalkSources_855236_9.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f726cf5884f62b409d7c52c07d0b939 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6c4af414-94f8-4128-a301-e0f25fc7d3cf_isrForCrosstalkSources_855238_53.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/6c4af414-94f8-4128-a301-e0f25fc7d3cf_isrForCrosstalkSources_855238_53.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd20e5207ba75e34f086d3135cf82f9 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72bea11b-04e8-4a21-8131-43b53d03bdaa_isrForCrosstalkSources_855240_47.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/72bea11b-04e8-4a21-8131-43b53d03bdaa_isrForCrosstalkSources_855240_47.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80f88fbe9d0ee2ca0fef28a555aaadb3 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fafd2f53-e79d-4935-95d8-fe0e43235cec_isrForCrosstalkSources_855239_50.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/fafd2f53-e79d-4935-95d8-fe0e43235cec_isrForCrosstalkSources_855239_50.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 161406d3e0059f6572a0f23938a09767 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f9a25a96-644d-46f5-9c67-fe1c08035ec8_isrForCrosstalkSources_855236_44.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/f9a25a96-644d-46f5-9c67-fe1c08035ec8_isrForCrosstalkSources_855236_44.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe2f84949b63f7bcebdc230e421d99a with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/eb6f3009-be63-4af1-9fe5-f27ab7ed2f1b_isrForCrosstalkSources_855241_21.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/eb6f3009-be63-4af1-9fe5-f27ab7ed2f1b_isrForCrosstalkSources_855241_21.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02b95ea5f06f1abe1d11134df79b95e2 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/54014cfd-1d6b-4320-a8ff-f40eae9d8582_isrForCrosstalkSources_855231_62.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/54014cfd-1d6b-4320-a8ff-f40eae9d8582_isrForCrosstalkSources_855231_62.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b95d9afba01c09f82be91eb340ff07 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d36b92eb-ea89-4946-9aa0-d3f772000c95_isrForCrosstalkSources_855155_11.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/d36b92eb-ea89-4946-9aa0-d3f772000c95_isrForCrosstalkSources_855155_11.stderr 2024-03-08 10:02:42 proc_lsst.multi:170 [INFO] [multi] job_id 16914792 proc_lsst.multi INFO: [multi] job_id 16914792 2024-03-08 10:02:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-08 10:02:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:42 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914792 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914792 parsl.executors.status_handling DEBUG: Launched block 171 on executor multi with job ID 16914792 parsl.executors.status_handling INFO: Allocated block ID 172 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-172 2024-03-08 10:02:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-172 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:42 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 115 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 115 2024-03-08 10:02:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=172 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb481d9ec03df739a5e5e7b00bebc893 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c2b06ab3-e5b0-430a-946e-49e2f8c48e0c_isrForCrosstalkSources_855241_61.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c2b06ab3-e5b0-430a-946e-49e2f8c48e0c_isrForCrosstalkSources_855241_61.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881e722f47e02fb2c6ae7245cbf61eef with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc11fb06-2ab8-47e2-82de-6a6ab143a61b_isrForCrosstalkSources_855234_51.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/fc11fb06-2ab8-47e2-82de-6a6ab143a61b_isrForCrosstalkSources_855234_51.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b7179e119698d354100fe4fab952cbd with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/209c5818-3f93-4b5d-adf3-2be2a545d239_isrForCrosstalkSources_855235_23.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/209c5818-3f93-4b5d-adf3-2be2a545d239_isrForCrosstalkSources_855235_23.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 505aa58914c78bddc247032940ea0f83 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d7340b05-6e11-4758-8bd2-781054a63a37_isrForCrosstalkSources_855241_40.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/d7340b05-6e11-4758-8bd2-781054a63a37_isrForCrosstalkSources_855241_40.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5c37802a135d6d35401e1289f3dfd49 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6317b03d-26c5-4b81-9f2a-eee46aa422e9_isrForCrosstalkSources_855242_52.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/6317b03d-26c5-4b81-9f2a-eee46aa422e9_isrForCrosstalkSources_855242_52.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379a91462428bcac1b04e87cfd1471e5 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/15409bfe-7ce6-4727-953f-a1bf77d67604_isrForCrosstalkSources_855233_39.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/15409bfe-7ce6-4727-953f-a1bf77d67604_isrForCrosstalkSources_855233_39.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cef577515b38796ce4c6a1c3ccf62107 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/25a7fcd5-befa-429b-8047-425863b0726d_isrForCrosstalkSources_855236_14.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/25a7fcd5-befa-429b-8047-425863b0726d_isrForCrosstalkSources_855236_14.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1890d21a3a3c96d86a813e51af5e94b9 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d305ce3-a7c3-4151-805c-bf877b40dc23_isrForCrosstalkSources_855237_60.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/9d305ce3-a7c3-4151-805c-bf877b40dc23_isrForCrosstalkSources_855237_60.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 708fdf72dd1fbc3ad2cf9ebca7ccd10d with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/02b08377-dcaa-47c2-8613-33b4d4dd092c_isrForCrosstalkSources_855232_23.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/02b08377-dcaa-47c2-8613-33b4d4dd092c_isrForCrosstalkSources_855232_23.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96b11b0eb1157312d70991e80424be2a with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3aad2da6-4a7e-48d3-ab8e-f47cce2cf095_isrForCrosstalkSources_855155_40.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/3aad2da6-4a7e-48d3-ab8e-f47cce2cf095_isrForCrosstalkSources_855155_40.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f547a326aefe9d3bbc4ca9b06180d42 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e4960ed0-8082-4307-9c46-b63ddf58b4bd_isrForCrosstalkSources_855238_61.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/e4960ed0-8082-4307-9c46-b63ddf58b4bd_isrForCrosstalkSources_855238_61.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c0ca4bf901ce147d982ae769e65f80c with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/534714b7-e337-4b62-ba01-5c9fb8b31414_isrForCrosstalkSources_855234_27.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/534714b7-e337-4b62-ba01-5c9fb8b31414_isrForCrosstalkSources_855234_27.stderr parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 870a81f0e76ea6f1c432a336e55a2cbd with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1554753b-5fed-47d3-8a26-f025b1f2cff9_isrForCrosstalkSources_855237_62.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/1554753b-5fed-47d3-8a26-f025b1f2cff9_isrForCrosstalkSources_855237_62.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c64ba49c4bdfc6385a0c03f07fde686d with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/de060991-6ef2-49a7-8cc7-9da4888e61c8_isrForCrosstalkSources_855237_15.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/de060991-6ef2-49a7-8cc7-9da4888e61c8_isrForCrosstalkSources_855237_15.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2afced481afde92a9276a4ffc0a37831 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a803e7fa-74db-445e-802f-8a37df69e422_isrForCrosstalkSources_855239_46.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a803e7fa-74db-445e-802f-8a37df69e422_isrForCrosstalkSources_855239_46.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c51bec1418bce3b05d611fe63ef7395b with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bb46a161-7a60-4678-a045-0e206f610b6c_isrForCrosstalkSources_855155_22.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bb46a161-7a60-4678-a045-0e206f610b6c_isrForCrosstalkSources_855155_22.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c8af3a3a869438de857c06816305295 with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/80c37bad-8d39-43df-b4a8-d78d099b881c_isrForCrosstalkSources_855237_35.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/80c37bad-8d39-43df-b4a8-d78d099b881c_isrForCrosstalkSources_855237_35.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c1c76aa425bee3eb9d3560c0921d546 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4733ab5b-b83c-4894-aa2d-79ccf1978292_isrForCrosstalkSources_855154_34.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/4733ab5b-b83c-4894-aa2d-79ccf1978292_isrForCrosstalkSources_855154_34.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80f052db270172a20cfa13817c70433 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ae5e31b5-899b-47d4-96db-cd1d4c151853_isrForCrosstalkSources_855237_61.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/ae5e31b5-899b-47d4-96db-cd1d4c151853_isrForCrosstalkSources_855237_61.stderr parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9cd88a35acca81aafcc8b001b71ca8e with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/edf91535-e1c5-418c-b58e-2ec5dfafb2a0_isrForCrosstalkSources_855154_52.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/edf91535-e1c5-418c-b58e-2ec5dfafb2a0_isrForCrosstalkSources_855154_52.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4639f578fed7eb562dded4dd48e993a1 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/baa03f1f-9b81-4c6d-ba8a-b5e85b53535c_isrForCrosstalkSources_855155_31.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/baa03f1f-9b81-4c6d-ba8a-b5e85b53535c_isrForCrosstalkSources_855155_31.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79506158b102ab061060d32f70628a52 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2bcde293-2936-4903-b36d-98569584065c_isrForCrosstalkSources_855239_14.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2bcde293-2936-4903-b36d-98569584065c_isrForCrosstalkSources_855239_14.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0bd5d9254d74cd900c8bf45fe978728 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9059a1b3-c5a4-48f1-8490-76745b0aa590_isrForCrosstalkSources_855239_49.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9059a1b3-c5a4-48f1-8490-76745b0aa590_isrForCrosstalkSources_855239_49.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69667697c0d1e64fe6516e398f656b38 with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bbd6520c-b81e-446c-9df8-2c4997c1b37b_isrForCrosstalkSources_855155_53.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/bbd6520c-b81e-446c-9df8-2c4997c1b37b_isrForCrosstalkSources_855155_53.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 640df3e6afe7c08a6c0b045da55932b3 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c9209bca-2068-4f38-85a9-db1aa5148b3d_isrForCrosstalkSources_855239_41.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/c9209bca-2068-4f38-85a9-db1aa5148b3d_isrForCrosstalkSources_855239_41.stderr parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c06c3dfcc48f344af483938f29edeef with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a75155b2-ee31-4f21-bc9e-1740c1212407_isrForCrosstalkSources_855239_52.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a75155b2-ee31-4f21-bc9e-1740c1212407_isrForCrosstalkSources_855239_52.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1104106df43597afa2b69e0254426866 with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/8dba948e-3a0e-4077-b60f-6a76acb2a14b_isrForCrosstalkSources_855235_47.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/8dba948e-3a0e-4077-b60f-6a76acb2a14b_isrForCrosstalkSources_855235_47.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 505a82860610b11510d19cf12f073ead with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4cc6360a-8282-433f-a960-28e00459bb46_isrForCrosstalkSources_855242_50.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/4cc6360a-8282-433f-a960-28e00459bb46_isrForCrosstalkSources_855242_50.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b94b12ec2a289592da2dd4b3c2e056c9 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3cc28b9-421b-4075-8e72-147149b2eb7e_isrForCrosstalkSources_855236_15.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/c3cc28b9-421b-4075-8e72-147149b2eb7e_isrForCrosstalkSources_855236_15.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7caa5ecd30b7736f0d922d291df31fd with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/397693f6-086a-4ce2-bb6c-9e65507bee18_isrForCrosstalkSources_855154_42.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/397693f6-086a-4ce2-bb6c-9e65507bee18_isrForCrosstalkSources_855154_42.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d293ec19da4c1a1045be7cb44c839f4b with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0df4bde1-c858-4373-ad1e-9222df325988_isrForCrosstalkSources_855155_54.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0df4bde1-c858-4373-ad1e-9222df325988_isrForCrosstalkSources_855155_54.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c963744b2539b46ea48980326e75497b with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/75e49a83-8204-42cd-b5ed-03d63f4866b9_isrForCrosstalkSources_855154_55.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/75e49a83-8204-42cd-b5ed-03d63f4866b9_isrForCrosstalkSources_855154_55.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b5adbb10f9d35a3aa4216fefc4cce20 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ddad05d8-23ea-48d2-b714-f2316005ee85_isrForCrosstalkSources_855231_41.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ddad05d8-23ea-48d2-b714-f2316005ee85_isrForCrosstalkSources_855231_41.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6479e2c606d0b36cb2c1c40e3ecf885 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0cf84850-5a63-404d-9368-5bb29221a472_isrForCrosstalkSources_855155_57.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/0cf84850-5a63-404d-9368-5bb29221a472_isrForCrosstalkSources_855155_57.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f855813c5816600577f2c43d9dad6eed with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4d2b5265-9ccb-4e8c-9041-5e1ad68752a0_isrForCrosstalkSources_855231_37.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/4d2b5265-9ccb-4e8c-9041-5e1ad68752a0_isrForCrosstalkSources_855231_37.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 532a782346f5124e2f5ee3d4bbc268f1 with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d9c6ab32-43c8-4af8-bc57-0ccdde322770_isrForCrosstalkSources_855234_4.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/d9c6ab32-43c8-4af8-bc57-0ccdde322770_isrForCrosstalkSources_855234_4.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e9dc660cc35e4b0bbf0917ed3d3ae6a with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80132e35-fe15-416f-925f-0be82fa289db_isrForCrosstalkSources_855235_62.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/80132e35-fe15-416f-925f-0be82fa289db_isrForCrosstalkSources_855235_62.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35f0936dfe385fe72ba4205dfc10a676 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4889c96f-5c53-4464-97b7-c14341867a1e_isrForCrosstalkSources_855236_31.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/4889c96f-5c53-4464-97b7-c14341867a1e_isrForCrosstalkSources_855236_31.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b28de24092b195616c088fbf5c180c with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5361931b-eff3-4631-82fe-5d9e05282b88_isrForCrosstalkSources_855233_61.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5361931b-eff3-4631-82fe-5d9e05282b88_isrForCrosstalkSources_855233_61.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4fa161e96a8f4125c6572acd6af439b with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c75aaacf-75e7-4b4c-a687-d371476b2c71_isrForCrosstalkSources_855240_26.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c75aaacf-75e7-4b4c-a687-d371476b2c71_isrForCrosstalkSources_855240_26.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd19b69710e9590e702043d85ec781d0 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5f3c49e2-e9a8-4db9-b985-0a6b1e4f8952_isrForCrosstalkSources_855238_31.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/5f3c49e2-e9a8-4db9-b985-0a6b1e4f8952_isrForCrosstalkSources_855238_31.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb92cb98bfe1c5a3255a85b4746144fd with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b24360ff-fded-4866-8d76-99a9dad103ba_isrForCrosstalkSources_855231_27.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b24360ff-fded-4866-8d76-99a9dad103ba_isrForCrosstalkSources_855231_27.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 677a17c96b6f79d04fc3971bf1d25c0d with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c257b6a1-5182-4ec5-8e37-00ab82a456dc_isrForCrosstalkSources_855232_58.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c257b6a1-5182-4ec5-8e37-00ab82a456dc_isrForCrosstalkSources_855232_58.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaa0391407cb951922a0d3ee11ad5ebb with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a8b0576-5c28-4620-9995-8c65054cae46_isrForCrosstalkSources_855235_7.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/2a8b0576-5c28-4620-9995-8c65054cae46_isrForCrosstalkSources_855235_7.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0e007e1daadb1d92575769f9b1d12e1 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d27848b1-b532-4944-829d-0cdb6375331c_isrForCrosstalkSources_855240_61.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/d27848b1-b532-4944-829d-0cdb6375331c_isrForCrosstalkSources_855240_61.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78fdabda0de433c91d57b1dcaa147f8f with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/841d5d28-0864-451e-bfe9-9c6a2d470e40_isrForCrosstalkSources_855233_3.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/841d5d28-0864-451e-bfe9-9c6a2d470e40_isrForCrosstalkSources_855233_3.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16dc58ad806a008f10bdf09e34b624a2 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/42393a68-e71b-4300-947f-76013505a75f_isrForCrosstalkSources_855241_32.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/42393a68-e71b-4300-947f-76013505a75f_isrForCrosstalkSources_855241_32.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0cfa0a6b1e95dd6ca2aee57c3b1857a with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/bbd12cf1-a484-42b6-92b5-6d8c1ced5173_isrForCrosstalkSources_855231_6.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/bbd12cf1-a484-42b6-92b5-6d8c1ced5173_isrForCrosstalkSources_855231_6.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaa13219bec94135a2cdd57b017a1c1e with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e0b86132-cbac-47d6-b392-f7592949606b_isrForCrosstalkSources_855237_37.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/e0b86132-cbac-47d6-b392-f7592949606b_isrForCrosstalkSources_855237_37.stderr 2024-03-08 10:02:44 proc_lsst.multi:170 [INFO] [multi] job_id 16914793 proc_lsst.multi INFO: [multi] job_id 16914793 2024-03-08 10:02:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-08 10:02:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914793 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914793 parsl.executors.status_handling DEBUG: Launched block 172 on executor multi with job ID 16914793 parsl.executors.status_handling INFO: Allocated block ID 173 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-173 2024-03-08 10:02:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-173 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 116 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 116 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=173 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:44 proc_lsst.multi:170 [INFO] [multi] job_id 16914794 proc_lsst.multi INFO: [multi] job_id 16914794 2024-03-08 10:02:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-08 10:02:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:44 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914794 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914794 parsl.executors.status_handling DEBUG: Launched block 173 on executor multi with job ID 16914794 parsl.executors.status_handling INFO: Allocated block ID 174 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-174 2024-03-08 10:02:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-174 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:44 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 117 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 117 2024-03-08 10:02:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=174 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0856dd3d05336e67ca42530d4b8324 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3117b793-d6d6-4a24-809d-1f6b158f1d49_isrForCrosstalkSources_855241_49.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3117b793-d6d6-4a24-809d-1f6b158f1d49_isrForCrosstalkSources_855241_49.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6128fe77a4474b0d796a59f00489e7c with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4f4d5892-bf95-4154-9a86-94dafaea5cd4_isrForCrosstalkSources_855240_8.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4f4d5892-bf95-4154-9a86-94dafaea5cd4_isrForCrosstalkSources_855240_8.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 533b3681835eb95f2e38af34fa6cfc2e with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/32eac1bf-5f44-48c9-947f-ba060e785917_isrForCrosstalkSources_855241_56.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/32eac1bf-5f44-48c9-947f-ba060e785917_isrForCrosstalkSources_855241_56.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 707b9ed9e7a4d8afe24bf9cce66d90aa with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9639e41f-daa6-40f2-ac21-922ab52fd7a7_isrForCrosstalkSources_855231_61.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/9639e41f-daa6-40f2-ac21-922ab52fd7a7_isrForCrosstalkSources_855231_61.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ddb4132ab76b50a237bfef08d5526f with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c5d6fb80-765c-4f3d-8b0f-799bd1fdcd80_isrForCrosstalkSources_855232_56.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/c5d6fb80-765c-4f3d-8b0f-799bd1fdcd80_isrForCrosstalkSources_855232_56.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39e37f389dc36d1cb11566114538c66d with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5175a26a-5c7e-49cf-974f-1073b33cc0c4_isrForCrosstalkSources_855233_15.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/5175a26a-5c7e-49cf-974f-1073b33cc0c4_isrForCrosstalkSources_855233_15.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8757c30101eca3f6f76837fb8e1b1593 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/32366752-1055-4a6f-b908-6986623803ca_isrForCrosstalkSources_855242_30.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/32366752-1055-4a6f-b908-6986623803ca_isrForCrosstalkSources_855242_30.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 805bad87124abcd96fb4fb64ec2fef67 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/424d13d9-6960-4498-91f1-ac709db922d9_isrForCrosstalkSources_855241_16.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/424d13d9-6960-4498-91f1-ac709db922d9_isrForCrosstalkSources_855241_16.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97a6f8b16213a30674a1229d937154c6 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d4a4efb6-26cf-4b4b-9e10-d91f208aed21_isrForCrosstalkSources_855156_40.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/d4a4efb6-26cf-4b4b-9e10-d91f208aed21_isrForCrosstalkSources_855156_40.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6cd1a9d8330b8df1e1f2cd7416f6fa2 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a3d46aeb-883d-46d9-ac14-4df4aa4cd093_isrForCrosstalkSources_855235_52.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a3d46aeb-883d-46d9-ac14-4df4aa4cd093_isrForCrosstalkSources_855235_52.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 109524a2cbb97a9579c01a87efe1301e with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d1201382-ef02-4b87-b365-4dd5a3755329_isrForCrosstalkSources_855235_53.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/d1201382-ef02-4b87-b365-4dd5a3755329_isrForCrosstalkSources_855235_53.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f40f63d1fee43c84e9e4087c4d64c60 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2995019e-40c9-4ff9-8d18-0ceeed740f52_isrForCrosstalkSources_855234_53.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/2995019e-40c9-4ff9-8d18-0ceeed740f52_isrForCrosstalkSources_855234_53.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ba2387e56ebecf8c576bcb5cef1d151 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/76918276-0d1e-4f1c-8081-d2eacbc4a65d_isrForCrosstalkSources_855231_17.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/76918276-0d1e-4f1c-8081-d2eacbc4a65d_isrForCrosstalkSources_855231_17.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d22c451efa3cc1c3988fb81ce59993e with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6295aa04-26f5-4e31-b274-147022823de4_isrForCrosstalkSources_855235_22.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/6295aa04-26f5-4e31-b274-147022823de4_isrForCrosstalkSources_855235_22.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090e5d43b3a921eaff12720e52efa32b with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c9146d2-97fe-4d9f-b8d9-6882e294b244_isrForCrosstalkSources_855241_54.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/7c9146d2-97fe-4d9f-b8d9-6882e294b244_isrForCrosstalkSources_855241_54.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b77bf17fc46c7f88e083869289d65bc0 with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3c03f5d1-2e27-42ca-b433-af7bda4b998d_isrForCrosstalkSources_855241_22.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/3c03f5d1-2e27-42ca-b433-af7bda4b998d_isrForCrosstalkSources_855241_22.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9de8d85dbca971ad1ea074aa9472ac8e with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/639be7b4-6dc7-4df8-86eb-039dbe39d80c_isrForCrosstalkSources_855239_45.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/639be7b4-6dc7-4df8-86eb-039dbe39d80c_isrForCrosstalkSources_855239_45.stderr 2024-03-08 10:02:46 proc_lsst.multi:170 [INFO] [multi] job_id 16914800 proc_lsst.multi INFO: [multi] job_id 16914800 2024-03-08 10:02:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-08 10:02:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914800 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914800 parsl.executors.status_handling DEBUG: Launched block 174 on executor multi with job ID 16914800 parsl.executors.status_handling INFO: Allocated block ID 175 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-175 2024-03-08 10:02:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-175 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 118 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 118 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=175 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e514098b387057791b6ef2e5042d6827 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d4db7c2b-1586-401e-82c3-ec3310cc66c9_isrForCrosstalkSources_855154_22.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/d4db7c2b-1586-401e-82c3-ec3310cc66c9_isrForCrosstalkSources_855154_22.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee921641ac6eb1a8d51c56e8284052c4 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/90e53691-4aa3-4e03-969d-adef868847b2_isrForCrosstalkSources_855242_2.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/90e53691-4aa3-4e03-969d-adef868847b2_isrForCrosstalkSources_855242_2.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad43df7aa35c16005c19ccbae0ff9217 with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c7b0f8b2-03e8-4449-898f-182108b30d75_isrForCrosstalkSources_855237_16.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/c7b0f8b2-03e8-4449-898f-182108b30d75_isrForCrosstalkSources_855237_16.stderr parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f122d202c2e3182166a7ed31d2e68eea with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3c242d9-9990-4680-9ee5-14a2dc487bb4_isrForCrosstalkSources_855235_57.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/f3c242d9-9990-4680-9ee5-14a2dc487bb4_isrForCrosstalkSources_855235_57.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bd1dc84e7b717e0cc7857f0e68ca194 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/04e3518b-dbba-4e18-81ff-4933195b672b_isrForCrosstalkSources_855156_22.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/04e3518b-dbba-4e18-81ff-4933195b672b_isrForCrosstalkSources_855156_22.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6d4b5508a929f1b0d17e3ee35c1b4f7 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de7aeb26-70c0-42f0-b306-f1a78a1fd7a2_isrForCrosstalkSources_855233_52.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/de7aeb26-70c0-42f0-b306-f1a78a1fd7a2_isrForCrosstalkSources_855233_52.stderr 2024-03-08 10:02:46 proc_lsst.multi:170 [INFO] [multi] job_id 16914801 proc_lsst.multi INFO: [multi] job_id 16914801 2024-03-08 10:02:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-08 10:02:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:46 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914801 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914801 parsl.executors.status_handling DEBUG: Launched block 175 on executor multi with job ID 16914801 parsl.executors.status_handling INFO: Allocated block ID 176 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-176 2024-03-08 10:02:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-176 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:46 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 119 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 119 2024-03-08 10:02:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=176 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20895f19b7ce86da6315338b73e31a0e with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ec196fdf-72ef-4a8e-87c2-e299064542fe_isrForCrosstalkSources_855235_59.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/ec196fdf-72ef-4a8e-87c2-e299064542fe_isrForCrosstalkSources_855235_59.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77b757a878714b76b4fdb3712443f53d with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ccf298f5-eb27-4d7a-ab82-a9a4c108c3a7_isrForCrosstalkSources_855231_9.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/ccf298f5-eb27-4d7a-ab82-a9a4c108c3a7_isrForCrosstalkSources_855231_9.stderr 2024-03-08 10:02:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914802 proc_lsst.multi INFO: [multi] job_id 16914802 2024-03-08 10:02:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-08 10:02:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914802 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914802 parsl.executors.status_handling DEBUG: Launched block 176 on executor multi with job ID 16914802 parsl.executors.status_handling INFO: Allocated block ID 177 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-177 2024-03-08 10:02:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-177 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 120 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 120 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=177 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914803 proc_lsst.multi INFO: [multi] job_id 16914803 2024-03-08 10:02:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-08 10:02:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914803 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914803 parsl.executors.status_handling DEBUG: Launched block 177 on executor multi with job ID 16914803 parsl.executors.status_handling INFO: Allocated block ID 178 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-178 2024-03-08 10:02:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-178 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 121 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 121 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=178 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914804 proc_lsst.multi INFO: [multi] job_id 16914804 2024-03-08 10:02:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-08 10:02:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914804 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914804 parsl.executors.status_handling DEBUG: Launched block 178 on executor multi with job ID 16914804 parsl.executors.status_handling INFO: Allocated block ID 179 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-179 2024-03-08 10:02:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-179 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 122 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 122 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=179 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a6a99dad166159e8814b64ecdf2db50 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d72747a4-6049-4231-b323-1215c521b619_isrForCrosstalkSources_855233_43.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/d72747a4-6049-4231-b323-1215c521b619_isrForCrosstalkSources_855233_43.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 176effe34cfeb08921c1eca234dc1609 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9f86ed45-7f63-4f41-a0d4-9887de11f536_isrForCrosstalkSources_855233_13.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/9f86ed45-7f63-4f41-a0d4-9887de11f536_isrForCrosstalkSources_855233_13.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8243f67a7682788ae1aaf0757696f05 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/869b5115-352e-4617-8575-ef7f092f5ec7_isrForCrosstalkSources_855154_36.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/869b5115-352e-4617-8575-ef7f092f5ec7_isrForCrosstalkSources_855154_36.stderr 2024-03-08 10:02:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914805 proc_lsst.multi INFO: [multi] job_id 16914805 2024-03-08 10:02:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-08 10:02:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914805 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914805 parsl.executors.status_handling DEBUG: Launched block 179 on executor multi with job ID 16914805 parsl.executors.status_handling INFO: Allocated block ID 180 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-180 2024-03-08 10:02:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-180 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 123 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 123 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=180 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0f8ccf346c89c8172e2c990a45c6960 with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/781649e4-c586-45da-a3ed-34c76c073105_isrForCrosstalkSources_855239_25.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/781649e4-c586-45da-a3ed-34c76c073105_isrForCrosstalkSources_855239_25.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99b35aed0abb0efa0b9d37109c30cedf with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/1b28ef59-77c3-4f0a-bad5-0450b4515f63_isrForCrosstalkSources_855242_14.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/1b28ef59-77c3-4f0a-bad5-0450b4515f63_isrForCrosstalkSources_855242_14.stderr 2024-03-08 10:02:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914806 proc_lsst.multi INFO: [multi] job_id 16914806 2024-03-08 10:02:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-08 10:02:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914806 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914806 parsl.executors.status_handling DEBUG: Launched block 180 on executor multi with job ID 16914806 parsl.executors.status_handling INFO: Allocated block ID 181 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-181 2024-03-08 10:02:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-181 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 124 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 124 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=181 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:47 proc_lsst.multi:170 [INFO] [multi] job_id 16914807 proc_lsst.multi INFO: [multi] job_id 16914807 2024-03-08 10:02:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-08 10:02:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914807 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914807 parsl.executors.status_handling DEBUG: Launched block 181 on executor multi with job ID 16914807 parsl.executors.status_handling INFO: Allocated block ID 182 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-182 2024-03-08 10:02:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-182 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:47 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 125 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 125 2024-03-08 10:02:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=182 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2b923167176602900896b4f55786e3 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/784f6bc2-f984-426a-8d2a-011caf81b476_isrForCrosstalkSources_855239_24.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/784f6bc2-f984-426a-8d2a-011caf81b476_isrForCrosstalkSources_855239_24.stderr 2024-03-08 10:02:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914808 proc_lsst.multi INFO: [multi] job_id 16914808 2024-03-08 10:02:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-08 10:02:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914808 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914808 parsl.executors.status_handling DEBUG: Launched block 182 on executor multi with job ID 16914808 parsl.executors.status_handling INFO: Allocated block ID 183 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-183 2024-03-08 10:02:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-183 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 126 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 126 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=183 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e94a1fc63c661c66e63932433c13d42 with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c97a9dfd-bddb-4915-9f2b-79eee77a3ecb_isrForCrosstalkSources_855240_35.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/c97a9dfd-bddb-4915-9f2b-79eee77a3ecb_isrForCrosstalkSources_855240_35.stderr 2024-03-08 10:02:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914809 proc_lsst.multi INFO: [multi] job_id 16914809 2024-03-08 10:02:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-08 10:02:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914809 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914809 parsl.executors.status_handling DEBUG: Launched block 183 on executor multi with job ID 16914809 parsl.executors.status_handling INFO: Allocated block ID 184 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-184 2024-03-08 10:02:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-184 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.dataflow.memoization DEBUG: Storing app cache entry 723250f755597a7147bf22aa99aeff07 with result from task 908 proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 127 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 127 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.dataflow.dflow DEBUG: Task 908 checkpointed proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=184 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9ffd4e91-19ad-48d8-b280-29eadc69e0e1_isrForCrosstalkSources_855239_3.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/9ffd4e91-19ad-48d8-b280-29eadc69e0e1_isrForCrosstalkSources_855239_3.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acbc3827ddfd42fc9895530c1dc6d113 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/34ab5a60-a8e2-4887-9195-d9d83982e3ad_isrForCrosstalkSources_855241_3.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/34ab5a60-a8e2-4887-9195-d9d83982e3ad_isrForCrosstalkSources_855241_3.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75b6dd29b4e18f857164bcbf4404ac5b with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ccba67e3-9e64-4100-9d00-04625a397736_isrForCrosstalkSources_855155_27.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/ccba67e3-9e64-4100-9d00-04625a397736_isrForCrosstalkSources_855155_27.stderr 2024-03-08 10:02:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914810 proc_lsst.multi INFO: [multi] job_id 16914810 2024-03-08 10:02:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-08 10:02:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914810 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914810 parsl.executors.status_handling DEBUG: Launched block 184 on executor multi with job ID 16914810 parsl.executors.status_handling INFO: Allocated block ID 185 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-185 2024-03-08 10:02:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-185 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 128 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 128 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=185 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88f69e8837e1b924f3c20e91ef5bd2fe with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5d1a0095-614c-4458-83fe-b9ead8fe0d13_isrForCrosstalkSources_855156_15.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/5d1a0095-614c-4458-83fe-b9ead8fe0d13_isrForCrosstalkSources_855156_15.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1c0fa9fa78b845ac2248505ba284f7 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/62efd87c-f514-408a-ad63-df779d7c7a81_isrForCrosstalkSources_855238_13.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/62efd87c-f514-408a-ad63-df779d7c7a81_isrForCrosstalkSources_855238_13.stderr parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 631ee183dfc996b8c58f2d7fbff25d94 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/884e2e18-ae29-4955-bfc2-73ee5ae4ad0b_isrForCrosstalkSources_855241_15.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/884e2e18-ae29-4955-bfc2-73ee5ae4ad0b_isrForCrosstalkSources_855241_15.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95df4301655c479637a163c2724098f1 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8f8215ad-bb0a-4635-85f7-3691446b3dcf_isrForCrosstalkSources_855233_8.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/8f8215ad-bb0a-4635-85f7-3691446b3dcf_isrForCrosstalkSources_855233_8.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f799185e7813a95759ee1e1cfe33964b with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a0546e93-5f21-4633-afe9-11045febc5de_isrForCrosstalkSources_855235_20.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/a0546e93-5f21-4633-afe9-11045febc5de_isrForCrosstalkSources_855235_20.stderr 2024-03-08 10:02:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914811 proc_lsst.multi INFO: [multi] job_id 16914811 2024-03-08 10:02:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-08 10:02:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914811 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914811 parsl.executors.status_handling DEBUG: Launched block 185 on executor multi with job ID 16914811 parsl.executors.status_handling INFO: Allocated block ID 186 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-186 2024-03-08 10:02:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-186 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 129 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 129 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=186 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 718f47968e91259911ce2f4a025de99f with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/8c77cd27-ddbf-4f48-9071-de3b70c86d0e_isrForCrosstalkSources_855242_36.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/8c77cd27-ddbf-4f48-9071-de3b70c86d0e_isrForCrosstalkSources_855242_36.stderr parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d851511d08e3c713316cf8ed966fa7fb with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9ae9f7e-161b-43d2-99b5-2d8c8ead88fa_isrForCrosstalkSources_855156_24.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/a9ae9f7e-161b-43d2-99b5-2d8c8ead88fa_isrForCrosstalkSources_855156_24.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8344568ad50015cb1ea2e725b0acb4b8 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2419a22d-8abe-4b99-b8bc-96d08abe028f_isrForCrosstalkSources_855239_38.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/2419a22d-8abe-4b99-b8bc-96d08abe028f_isrForCrosstalkSources_855239_38.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0011f09f0754aa7e86f302917ada76e with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/9bbed066-1fab-4862-8965-cad677d0a454_isrForCrosstalkSources_855155_33.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/9bbed066-1fab-4862-8965-cad677d0a454_isrForCrosstalkSources_855155_33.stderr 2024-03-08 10:02:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914812 proc_lsst.multi INFO: [multi] job_id 16914812 2024-03-08 10:02:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-08 10:02:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914812 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914812 parsl.executors.status_handling DEBUG: Launched block 186 on executor multi with job ID 16914812 parsl.executors.status_handling INFO: Allocated block ID 187 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-187 2024-03-08 10:02:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-187 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 130 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 130 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=187 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e465368afb2726c4b4f2e86c51d89fbf with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/744ff8da-11b0-45f0-b4eb-6d37956837e7_isrForCrosstalkSources_855232_49.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/744ff8da-11b0-45f0-b4eb-6d37956837e7_isrForCrosstalkSources_855232_49.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a0e65924fab16c2848afabf9262a75 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/67000f2d-142a-4a17-9c1d-8360bcc11e8e_isrForCrosstalkSources_855237_48.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/67000f2d-142a-4a17-9c1d-8360bcc11e8e_isrForCrosstalkSources_855237_48.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ffd628e50504aee0a53a0ad90279b8d with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/3b1a10cf-a548-4080-8349-e37ba090e3ad_isrForCrosstalkSources_855242_27.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/3b1a10cf-a548-4080-8349-e37ba090e3ad_isrForCrosstalkSources_855242_27.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21b86b4a4c0bf793e86697da97761a96 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ef5b24ec-abb8-4761-a8bb-f4123b62a446_isrForCrosstalkSources_855242_51.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/ef5b24ec-abb8-4761-a8bb-f4123b62a446_isrForCrosstalkSources_855242_51.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb2aac6e4fca42a04c70b8f4069d2192 with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/1e9de8e0-2769-46c4-bc03-566a4b79c177_isrForCrosstalkSources_855236_21.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/1e9de8e0-2769-46c4-bc03-566a4b79c177_isrForCrosstalkSources_855236_21.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 280c71f248b07709dc69bbd92a2a1690 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c2afd191-016f-4fe9-8ffd-c7aceca1b024_isrForCrosstalkSources_855242_40.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/c2afd191-016f-4fe9-8ffd-c7aceca1b024_isrForCrosstalkSources_855242_40.stderr 2024-03-08 10:02:48 proc_lsst.multi:170 [INFO] [multi] job_id 16914813 proc_lsst.multi INFO: [multi] job_id 16914813 2024-03-08 10:02:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-08 10:02:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914813 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914813 parsl.executors.status_handling DEBUG: Launched block 187 on executor multi with job ID 16914813 parsl.executors.status_handling INFO: Allocated block ID 188 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-188 2024-03-08 10:02:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-188 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:48 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 131 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 131 2024-03-08 10:02:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=188 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b45a2a56bf5df28d04477b250ea46985 with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1e50c119-9a73-4a0d-9c8b-4619980b9f02_isrForCrosstalkSources_855239_30.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/1e50c119-9a73-4a0d-9c8b-4619980b9f02_isrForCrosstalkSources_855239_30.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a410a828f5ba88c22b9ae1f55f2cd28d with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b45a4dc0-5d59-4ec6-8611-78ad8d145138_isrForCrosstalkSources_855233_30.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/b45a4dc0-5d59-4ec6-8611-78ad8d145138_isrForCrosstalkSources_855233_30.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry faf966ef623a27235e9464d8d30c044d with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/22b60404-cba1-48e2-9218-24061b3d0868_isrForCrosstalkSources_855156_14.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/22b60404-cba1-48e2-9218-24061b3d0868_isrForCrosstalkSources_855156_14.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58aca6dd8162ec376ea53a4477279640 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/32df5d95-0335-4ff6-b235-5f4cac34d027_isrForCrosstalkSources_855156_27.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/32df5d95-0335-4ff6-b235-5f4cac34d027_isrForCrosstalkSources_855156_27.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 012ce66cf6a80de6d6a408d516858222 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b75c0d10-6280-4979-b14a-5fbc4e14ebcb_isrForCrosstalkSources_855241_14.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/b75c0d10-6280-4979-b14a-5fbc4e14ebcb_isrForCrosstalkSources_855241_14.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4addff8cc43b6d9c9f017e7a8907de with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/04de2aac-bc8e-4647-8c8d-b09aa559155a_isrForCrosstalkSources_855238_9.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/04de2aac-bc8e-4647-8c8d-b09aa559155a_isrForCrosstalkSources_855238_9.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bd8b1f2bf3c87e24bdc53c817846bdc with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9d2d872-807b-4cb1-86e9-8b55ea7f11df_isrForCrosstalkSources_855231_3.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855231/b9d2d872-807b-4cb1-86e9-8b55ea7f11df_isrForCrosstalkSources_855231_3.stderr parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e03db13b2402c6151564539f0a4bb75 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2ad0345-0675-4c1f-9891-78ecacc036c4_isrForCrosstalkSources_855239_40.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855239/a2ad0345-0675-4c1f-9891-78ecacc036c4_isrForCrosstalkSources_855239_40.stderr 2024-03-08 10:02:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914814 proc_lsst.multi INFO: [multi] job_id 16914814 2024-03-08 10:02:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-08 10:02:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914814 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914814 parsl.executors.status_handling DEBUG: Launched block 188 on executor multi with job ID 16914814 parsl.executors.status_handling INFO: Allocated block ID 189 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-189 2024-03-08 10:02:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-189 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 132 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 132 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=189 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Storing app cache entry f0c389cb0ebaa044f55c4f629aa5f6f4 with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/000ae0b2-f965-433a-9b25-808b14a70e04_isrForCrosstalkSources_855242_37.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/000ae0b2-f965-433a-9b25-808b14a70e04_isrForCrosstalkSources_855242_37.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c98691e953a58c06ee5f6b98150fb755 with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/fa651f48-51bb-40e3-bdd6-e9be5f861044_isrForCrosstalkSources_855242_34.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855242/fa651f48-51bb-40e3-bdd6-e9be5f861044_isrForCrosstalkSources_855242_34.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9fe5acc94ca0e34f2fbe6013721bd96 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7d3d3c09-0649-40e2-95a3-550848722858_isrForCrosstalkSources_855232_18.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/7d3d3c09-0649-40e2-95a3-550848722858_isrForCrosstalkSources_855232_18.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43ab4ca98c7cf5045b5a61e9dfc9423d with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/461a93f5-30b4-4b8a-b6d6-10166bcae352_isrForCrosstalkSources_855156_11.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/461a93f5-30b4-4b8a-b6d6-10166bcae352_isrForCrosstalkSources_855156_11.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd4e941ff16afd54e2e8c2ce7cf78c9e with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/78cb3f40-51f9-4705-bcf5-005e4318cf6b_isrForCrosstalkSources_855240_17.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/78cb3f40-51f9-4705-bcf5-005e4318cf6b_isrForCrosstalkSources_855240_17.stderr 2024-03-08 10:02:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914815 proc_lsst.multi INFO: [multi] job_id 16914815 2024-03-08 10:02:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-08 10:02:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914815 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914815 parsl.executors.status_handling DEBUG: Launched block 189 on executor multi with job ID 16914815 parsl.executors.status_handling INFO: Allocated block ID 190 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-190 2024-03-08 10:02:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-190 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 133 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 133 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=190 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53fb71d992d9660328dc2b62a5eea3ba with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/2b75123a-3afb-4377-b4a1-c04fa3a8ac23_isrForCrosstalkSources_855237_1.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/2b75123a-3afb-4377-b4a1-c04fa3a8ac23_isrForCrosstalkSources_855237_1.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9b186627934244e22b0b5c4e3ee77d2 with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4a1c5398-ce76-488f-953d-dfa6a7dbdb00_isrForCrosstalkSources_855240_44.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/4a1c5398-ce76-488f-953d-dfa6a7dbdb00_isrForCrosstalkSources_855240_44.stderr 2024-03-08 10:02:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914816 proc_lsst.multi INFO: [multi] job_id 16914816 2024-03-08 10:02:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-08 10:02:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914816 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914816 parsl.executors.status_handling DEBUG: Launched block 190 on executor multi with job ID 16914816 parsl.executors.status_handling INFO: Allocated block ID 191 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-191 2024-03-08 10:02:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-191 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 134 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 134 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=191 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5d6fb2c823896648f313897d936586e with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/42ff4bce-fd5d-4536-93cb-0c6ad613bdc4_isrForCrosstalkSources_855236_18.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855236/42ff4bce-fd5d-4536-93cb-0c6ad613bdc4_isrForCrosstalkSources_855236_18.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba4137576f3c30ab83434b1925de959d with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/39497209-0e56-4755-b056-48164221b4d6_isrForCrosstalkSources_855154_49.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/39497209-0e56-4755-b056-48164221b4d6_isrForCrosstalkSources_855154_49.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6acf77cf8007846c3484ba7470b434e2 with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c3dd3a57-deda-4c9a-ab31-3ae0ec5ac26f_isrForCrosstalkSources_855156_3.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/c3dd3a57-deda-4c9a-ab31-3ae0ec5ac26f_isrForCrosstalkSources_855156_3.stderr 2024-03-08 10:02:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914817 proc_lsst.multi INFO: [multi] job_id 16914817 2024-03-08 10:02:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-08 10:02:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914817 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914817 parsl.executors.status_handling DEBUG: Launched block 191 on executor multi with job ID 16914817 parsl.executors.status_handling INFO: Allocated block ID 192 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-192 2024-03-08 10:02:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-192 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 135 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 135 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=192 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 445b56ef8b7044b993615380be5eda9d with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e8e3cdb4-cde4-49f7-8478-883d06e90fbe_isrForCrosstalkSources_855154_43.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855154/e8e3cdb4-cde4-49f7-8478-883d06e90fbe_isrForCrosstalkSources_855154_43.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 046ed12c3ac1837f68dc1301ab0c9aae with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/f58dbc77-4ac3-4d3b-8d81-4833c77a1da2_isrForCrosstalkSources_855237_42.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/f58dbc77-4ac3-4d3b-8d81-4833c77a1da2_isrForCrosstalkSources_855237_42.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec21ebbe5b7f3cc41b385a8cc609b396 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d8f6099a-69f4-46b2-97cc-ce746279a1c9_isrForCrosstalkSources_855237_43.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/d8f6099a-69f4-46b2-97cc-ce746279a1c9_isrForCrosstalkSources_855237_43.stderr 2024-03-08 10:02:49 proc_lsst.multi:170 [INFO] [multi] job_id 16914818 proc_lsst.multi INFO: [multi] job_id 16914818 2024-03-08 10:02:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-08 10:02:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914818 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914818 parsl.executors.status_handling DEBUG: Launched block 192 on executor multi with job ID 16914818 parsl.executors.status_handling INFO: Allocated block ID 193 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-193 2024-03-08 10:02:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-193 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:49 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 136 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 136 2024-03-08 10:02:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=193 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f2a1191deb1fbcd043d566d803fd16b with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1f3757d0-b725-4b48-88ae-36be11d82590_isrForCrosstalkSources_855240_49.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/1f3757d0-b725-4b48-88ae-36be11d82590_isrForCrosstalkSources_855240_49.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5a9bc64bef61c8e58204a50f2bb4eb4 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2b5c483b-699f-4eeb-82f0-2f6049a35898_isrForCrosstalkSources_855240_55.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/2b5c483b-699f-4eeb-82f0-2f6049a35898_isrForCrosstalkSources_855240_55.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ac18733b43678d00012575dbbc42511 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6ab13898-08ea-44fd-9b8f-2c3a71ccb79b_isrForCrosstalkSources_855232_50.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/6ab13898-08ea-44fd-9b8f-2c3a71ccb79b_isrForCrosstalkSources_855232_50.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f467a058f33eb16a1a8d5cf664f47ed7 with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bada7eae-9b21-43e3-b403-8479f55da098_isrForCrosstalkSources_855237_58.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855237/bada7eae-9b21-43e3-b403-8479f55da098_isrForCrosstalkSources_855237_58.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06329364db669de129a0997cd36254ee with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/23df32e9-8371-4e6c-9ac9-39bb09904c73_isrForCrosstalkSources_855232_37.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/23df32e9-8371-4e6c-9ac9-39bb09904c73_isrForCrosstalkSources_855232_37.stderr parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0128389b433c70cf7d183110125744c with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/0ed09341-f3b3-4864-893f-4bba908574a1_isrForCrosstalkSources_855233_47.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/0ed09341-f3b3-4864-893f-4bba908574a1_isrForCrosstalkSources_855233_47.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24d91ce99dffcfbe4a3855e32e46d8e9 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d6ff70fa-bd01-44a4-b20b-bd85da6619b1_isrForCrosstalkSources_855238_6.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855238/d6ff70fa-bd01-44a4-b20b-bd85da6619b1_isrForCrosstalkSources_855238_6.stderr 2024-03-08 10:02:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914819 proc_lsst.multi INFO: [multi] job_id 16914819 2024-03-08 10:02:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-08 10:02:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914819 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914819 parsl.executors.status_handling DEBUG: Launched block 193 on executor multi with job ID 16914819 parsl.executors.status_handling INFO: Allocated block ID 194 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-194 2024-03-08 10:02:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-194 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 137 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 137 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=194 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b2c712fd66ded6397d0ab5a9ead27f8 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a9c36e0f-a88e-492b-a179-5d226d6b1df4_isrForCrosstalkSources_855234_30.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/a9c36e0f-a88e-492b-a179-5d226d6b1df4_isrForCrosstalkSources_855234_30.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da893464d4ba393bc924ee63bfc20dfd with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/797b4129-cdb2-4369-9590-44b46942242b_isrForCrosstalkSources_855234_31.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855234/797b4129-cdb2-4369-9590-44b46942242b_isrForCrosstalkSources_855234_31.stderr 2024-03-08 10:02:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914820 proc_lsst.multi INFO: [multi] job_id 16914820 2024-03-08 10:02:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-08 10:02:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914820 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914820 parsl.executors.status_handling DEBUG: Launched block 194 on executor multi with job ID 16914820 parsl.executors.status_handling INFO: Allocated block ID 195 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-195 2024-03-08 10:02:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-195 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 138 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 138 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=195 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914821 proc_lsst.multi INFO: [multi] job_id 16914821 2024-03-08 10:02:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-08 10:02:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914821 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914821 parsl.executors.status_handling DEBUG: Launched block 195 on executor multi with job ID 16914821 parsl.executors.status_handling INFO: Allocated block ID 196 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-196 2024-03-08 10:02:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-196 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 139 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 139 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=196 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914822 proc_lsst.multi INFO: [multi] job_id 16914822 2024-03-08 10:02:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-08 10:02:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914822 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914822 parsl.executors.status_handling DEBUG: Launched block 196 on executor multi with job ID 16914822 parsl.executors.status_handling INFO: Allocated block ID 197 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-197 2024-03-08 10:02:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-197 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 140 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 140 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=197 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77a7a7e4331dfb9f4ce5e9796f52fb3c with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/ec26d339-043f-4eab-9421-4cf628f70196_isrForCrosstalkSources_855156_4.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855156/ec26d339-043f-4eab-9421-4cf628f70196_isrForCrosstalkSources_855156_4.stderr 2024-03-08 10:02:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914823 proc_lsst.multi INFO: [multi] job_id 16914823 2024-03-08 10:02:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-08 10:02:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914823 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914823 parsl.executors.status_handling DEBUG: Launched block 197 on executor multi with job ID 16914823 parsl.executors.status_handling INFO: Allocated block ID 198 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-198 2024-03-08 10:02:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-198 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 141 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 141 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=198 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adfc3ad422adfbe9824488db86da0f5f with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f0d6ee1d-0caf-462b-8030-1f4325547d94_isrForCrosstalkSources_855232_8.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855232/f0d6ee1d-0caf-462b-8030-1f4325547d94_isrForCrosstalkSources_855232_8.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5069949af7b529eb8320dc9b2c62e644 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/dcf4572c-7cf9-47e7-bee7-e0af51a1dfb6_isrForCrosstalkSources_855233_58.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855233/dcf4572c-7cf9-47e7-bee7-e0af51a1dfb6_isrForCrosstalkSources_855233_58.stderr 2024-03-08 10:02:50 proc_lsst.multi:170 [INFO] [multi] job_id 16914824 proc_lsst.multi INFO: [multi] job_id 16914824 2024-03-08 10:02:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-08 10:02:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914824 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914824 parsl.executors.status_handling DEBUG: Launched block 198 on executor multi with job ID 16914824 parsl.executors.status_handling INFO: Allocated block ID 199 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-199 2024-03-08 10:02:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-199 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:50 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 142 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 142 2024-03-08 10:02:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=199 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914825 proc_lsst.multi INFO: [multi] job_id 16914825 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914825 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914825 parsl.executors.status_handling DEBUG: Launched block 199 on executor multi with job ID 16914825 parsl.executors.status_handling INFO: Allocated block ID 200 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-200 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-200 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 143 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 143 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=200 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c148b031e74a7172cd8c5f3bd962b423 with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c834ab62-0749-445e-ae5b-d65908257c05_isrForCrosstalkSources_855241_42.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/c834ab62-0749-445e-ae5b-d65908257c05_isrForCrosstalkSources_855241_42.stderr 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914826 proc_lsst.multi INFO: [multi] job_id 16914826 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914826 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914826 parsl.executors.status_handling DEBUG: Launched block 200 on executor multi with job ID 16914826 parsl.executors.status_handling INFO: Allocated block ID 201 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-201 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-201 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 144 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 144 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=201 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914827 proc_lsst.multi INFO: [multi] job_id 16914827 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914827 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914827 parsl.executors.status_handling DEBUG: Launched block 201 on executor multi with job ID 16914827 parsl.executors.status_handling INFO: Allocated block ID 202 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-202 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-202 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 145 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 145 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=202 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914828 proc_lsst.multi INFO: [multi] job_id 16914828 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914828 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914828 parsl.executors.status_handling DEBUG: Launched block 202 on executor multi with job ID 16914828 parsl.executors.status_handling INFO: Allocated block ID 203 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-203 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-203 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 146 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 146 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=203 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914829 proc_lsst.multi INFO: [multi] job_id 16914829 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914829 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914829 parsl.executors.status_handling DEBUG: Launched block 203 on executor multi with job ID 16914829 parsl.executors.status_handling INFO: Allocated block ID 204 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-204 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-204 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 147 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 147 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=204 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e0f141e7758332cccc27443b56b561c with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7d1b7a04-dbef-4f61-afa5-d10254b0bc00_isrForCrosstalkSources_855155_62.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855155/7d1b7a04-dbef-4f61-afa5-d10254b0bc00_isrForCrosstalkSources_855155_62.stderr parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e6d4d97198430456c106a72fda03324 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b82c7b13-61ac-42b9-94d9-6e2e2438113a_isrForCrosstalkSources_855235_60.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855235/b82c7b13-61ac-42b9-94d9-6e2e2438113a_isrForCrosstalkSources_855235_60.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6dbb96206b8274966d2365b12d8ca94 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/20d03dd8-4222-4fab-b7f8-c047900d59cb_isrForCrosstalkSources_855240_10.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855240/20d03dd8-4222-4fab-b7f8-c047900d59cb_isrForCrosstalkSources_855240_10.stderr 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914830 proc_lsst.multi INFO: [multi] job_id 16914830 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914830 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914830 parsl.executors.status_handling DEBUG: Launched block 204 on executor multi with job ID 16914830 parsl.executors.status_handling INFO: Allocated block ID 205 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-205 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-205 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 148 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 148 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=205 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:51 proc_lsst.multi:170 [INFO] [multi] job_id 16914831 proc_lsst.multi INFO: [multi] job_id 16914831 2024-03-08 10:02:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-08 10:02:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914831 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914831 parsl.executors.status_handling DEBUG: Launched block 205 on executor multi with job ID 16914831 parsl.executors.status_handling INFO: Allocated block ID 206 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-206 2024-03-08 10:02:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-206 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:51 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 149 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 149 2024-03-08 10:02:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=206 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914832 proc_lsst.multi INFO: [multi] job_id 16914832 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914832 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914832 parsl.executors.status_handling DEBUG: Launched block 206 on executor multi with job ID 16914832 parsl.executors.status_handling INFO: Allocated block ID 207 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-207 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-207 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 150 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 150 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=207 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c9d6b03f021ce89aec2f83acf902e2e with result from task 830 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/017308e7-9d76-44a6-95e3-cfbdd32869f8_isrForCrosstalkSources_855241_57.stdout parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/logs/isrForCrosstalkSources/855241/017308e7-9d76-44a6-95e3-cfbdd32869f8_isrForCrosstalkSources_855241_57.stderr parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 930 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914833 proc_lsst.multi INFO: [multi] job_id 16914833 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914833 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914833 parsl.executors.status_handling DEBUG: Launched block 207 on executor multi with job ID 16914833 parsl.executors.status_handling INFO: Allocated block ID 208 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-208 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-208 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 151 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 151 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=208 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914834 proc_lsst.multi INFO: [multi] job_id 16914834 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914834 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914834 parsl.executors.status_handling DEBUG: Launched block 208 on executor multi with job ID 16914834 parsl.executors.status_handling INFO: Allocated block ID 209 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-209 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-209 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 152 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 152 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=209 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914835 proc_lsst.multi INFO: [multi] job_id 16914835 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914835 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914835 parsl.executors.status_handling DEBUG: Launched block 209 on executor multi with job ID 16914835 parsl.executors.status_handling INFO: Allocated block ID 210 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-210 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-210 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 153 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 153 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=210 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914836 proc_lsst.multi INFO: [multi] job_id 16914836 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914836 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914836 parsl.executors.status_handling DEBUG: Launched block 210 on executor multi with job ID 16914836 parsl.executors.status_handling INFO: Allocated block ID 211 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-211 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-211 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 154 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 154 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=211 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914837 proc_lsst.multi INFO: [multi] job_id 16914837 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914837 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914837 parsl.executors.status_handling DEBUG: Launched block 211 on executor multi with job ID 16914837 parsl.executors.status_handling INFO: Allocated block ID 212 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-212 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-212 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 155 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 155 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=212 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:52 proc_lsst.multi:170 [INFO] [multi] job_id 16914838 proc_lsst.multi INFO: [multi] job_id 16914838 2024-03-08 10:02:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-08 10:02:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914838 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914838 parsl.executors.status_handling DEBUG: Launched block 212 on executor multi with job ID 16914838 parsl.executors.status_handling INFO: Allocated block ID 213 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-213 2024-03-08 10:02:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-213 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:52 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 156 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 156 2024-03-08 10:02:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=213 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914839 proc_lsst.multi INFO: [multi] job_id 16914839 2024-03-08 10:02:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-08 10:02:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914839 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914839 parsl.executors.status_handling DEBUG: Launched block 213 on executor multi with job ID 16914839 parsl.executors.status_handling INFO: Allocated block ID 214 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-214 2024-03-08 10:02:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-214 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 157 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 157 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=214 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914840 proc_lsst.multi INFO: [multi] job_id 16914840 2024-03-08 10:02:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-08 10:02:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914840 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914840 parsl.executors.status_handling DEBUG: Launched block 214 on executor multi with job ID 16914840 parsl.executors.status_handling INFO: Allocated block ID 215 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-215 2024-03-08 10:02:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-215 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 158 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 158 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=215 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914841 proc_lsst.multi INFO: [multi] job_id 16914841 2024-03-08 10:02:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-08 10:02:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914841 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914841 parsl.executors.status_handling DEBUG: Launched block 215 on executor multi with job ID 16914841 parsl.executors.status_handling INFO: Allocated block ID 216 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-216 2024-03-08 10:02:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-216 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 159 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 159 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=216 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914842 proc_lsst.multi INFO: [multi] job_id 16914842 2024-03-08 10:02:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-08 10:02:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914842 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914842 parsl.executors.status_handling DEBUG: Launched block 216 on executor multi with job ID 16914842 parsl.executors.status_handling INFO: Allocated block ID 217 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-217 2024-03-08 10:02:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-217 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 160 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 160 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=217 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914843 proc_lsst.multi INFO: [multi] job_id 16914843 2024-03-08 10:02:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-08 10:02:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914843 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914843 parsl.executors.status_handling DEBUG: Launched block 217 on executor multi with job ID 16914843 parsl.executors.status_handling INFO: Allocated block ID 218 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-218 2024-03-08 10:02:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-218 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 161 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 161 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=218 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:53 proc_lsst.multi:170 [INFO] [multi] job_id 16914844 proc_lsst.multi INFO: [multi] job_id 16914844 2024-03-08 10:02:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-08 10:02:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914844 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914844 parsl.executors.status_handling DEBUG: Launched block 218 on executor multi with job ID 16914844 parsl.executors.status_handling INFO: Allocated block ID 219 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-219 2024-03-08 10:02:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-219 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 162 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 162 2024-03-08 10:02:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=219 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914845 proc_lsst.multi INFO: [multi] job_id 16914845 2024-03-08 10:02:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-08 10:02:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914845 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914845 parsl.executors.status_handling DEBUG: Launched block 219 on executor multi with job ID 16914845 parsl.executors.status_handling INFO: Allocated block ID 220 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-220 2024-03-08 10:02:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-220 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 163 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 163 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=220 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914846 proc_lsst.multi INFO: [multi] job_id 16914846 2024-03-08 10:02:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-08 10:02:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914846 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914846 parsl.executors.status_handling DEBUG: Launched block 220 on executor multi with job ID 16914846 parsl.executors.status_handling INFO: Allocated block ID 221 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-221 2024-03-08 10:02:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-221 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 164 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 164 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=221 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914847 proc_lsst.multi INFO: [multi] job_id 16914847 2024-03-08 10:02:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-08 10:02:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914847 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914847 parsl.executors.status_handling DEBUG: Launched block 221 on executor multi with job ID 16914847 parsl.executors.status_handling INFO: Allocated block ID 222 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-222 2024-03-08 10:02:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-222 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 165 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 165 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=222 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914848 proc_lsst.multi INFO: [multi] job_id 16914848 2024-03-08 10:02:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-08 10:02:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914848 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914848 parsl.executors.status_handling DEBUG: Launched block 222 on executor multi with job ID 16914848 parsl.executors.status_handling INFO: Allocated block ID 223 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-223 2024-03-08 10:02:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-223 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 166 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 166 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=223 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914849 proc_lsst.multi INFO: [multi] job_id 16914849 2024-03-08 10:02:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-08 10:02:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914849 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914849 parsl.executors.status_handling DEBUG: Launched block 223 on executor multi with job ID 16914849 parsl.executors.status_handling INFO: Allocated block ID 224 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-224 2024-03-08 10:02:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-224 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 167 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 167 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=224 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:54 proc_lsst.multi:170 [INFO] [multi] job_id 16914850 proc_lsst.multi INFO: [multi] job_id 16914850 2024-03-08 10:02:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-08 10:02:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914850 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914850 parsl.executors.status_handling DEBUG: Launched block 224 on executor multi with job ID 16914850 parsl.executors.status_handling INFO: Allocated block ID 225 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-225 2024-03-08 10:02:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-225 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 168 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 168 2024-03-08 10:02:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=225 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914851 proc_lsst.multi INFO: [multi] job_id 16914851 2024-03-08 10:02:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-08 10:02:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914851 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914851 parsl.executors.status_handling DEBUG: Launched block 225 on executor multi with job ID 16914851 parsl.executors.status_handling INFO: Allocated block ID 226 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-226 2024-03-08 10:02:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-226 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 169 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 169 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=226 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914852 proc_lsst.multi INFO: [multi] job_id 16914852 2024-03-08 10:02:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-08 10:02:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914852 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914852 parsl.executors.status_handling DEBUG: Launched block 226 on executor multi with job ID 16914852 parsl.executors.status_handling INFO: Allocated block ID 227 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-227 2024-03-08 10:02:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-227 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 170 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 170 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=227 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914853 proc_lsst.multi INFO: [multi] job_id 16914853 2024-03-08 10:02:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-08 10:02:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914853 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914853 parsl.executors.status_handling DEBUG: Launched block 227 on executor multi with job ID 16914853 parsl.executors.status_handling INFO: Allocated block ID 228 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-228 2024-03-08 10:02:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-228 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 171 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 171 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=228 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914854 proc_lsst.multi INFO: [multi] job_id 16914854 2024-03-08 10:02:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-08 10:02:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914854 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914854 parsl.executors.status_handling DEBUG: Launched block 228 on executor multi with job ID 16914854 parsl.executors.status_handling INFO: Allocated block ID 229 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-229 2024-03-08 10:02:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-229 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 172 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 172 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=229 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914855 proc_lsst.multi INFO: [multi] job_id 16914855 2024-03-08 10:02:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-08 10:02:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914855 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914855 parsl.executors.status_handling DEBUG: Launched block 229 on executor multi with job ID 16914855 parsl.executors.status_handling INFO: Allocated block ID 230 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-230 2024-03-08 10:02:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-230 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 173 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 173 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=230 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:55 proc_lsst.multi:170 [INFO] [multi] job_id 16914856 proc_lsst.multi INFO: [multi] job_id 16914856 2024-03-08 10:02:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-08 10:02:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914856 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914856 parsl.executors.status_handling DEBUG: Launched block 230 on executor multi with job ID 16914856 parsl.executors.status_handling INFO: Allocated block ID 231 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-231 2024-03-08 10:02:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-231 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 174 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 174 2024-03-08 10:02:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=231 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914857 proc_lsst.multi INFO: [multi] job_id 16914857 2024-03-08 10:02:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-08 10:02:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914857 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914857 parsl.executors.status_handling DEBUG: Launched block 231 on executor multi with job ID 16914857 parsl.executors.status_handling INFO: Allocated block ID 232 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-232 2024-03-08 10:02:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-232 2024-03-08 10:02:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-08 10:02:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-08 10:02:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-08 10:02:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-08 10:02:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-08 10:02:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-08 10:02:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-08 10:02:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-08 10:02:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-08 10:02:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-08 10:02:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 175 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 175 2024-03-08 10:02:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,n3064,198.48.92.26,10.64.129.64,169.254.95.120,10.64.65.64 -p 0 -c 1.0 -m None --poll 10 --task_port=54874 --result_port=54302 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190504/A1c/science#step0/20240308T100021Z/runinfo/000/multi --block_id=232 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-08 10:02:56 proc_lsst.multi:170 [INFO] [multi] job_id 16914858 proc_lsst.multi INFO: [multi] job_id 16914858 2024-03-08 10:02:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 176 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 176 2024-03-08 10:02:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-08 10:02:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16914858 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16914858 parsl.executors.status_handling DEBUG: Launched block 232 on executor multi with job ID 16914858 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23265528657488 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=232, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 83 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 53a915e73aaa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 53a915e73aaa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e6ffcfa327a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e6ffcfa327a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7c80f97a648a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7c80f97a648a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: acc89c3f2e81 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: acc89c3f2e81 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da1e915e2fe5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da1e915e2fe5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b8c337707e59 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b8c337707e59 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 025ecafe2fdb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 025ecafe2fdb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fbbec52e8bd0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fbbec52e8bd0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b069990b6d0b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b069990b6d0b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b8825e330d70 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b8825e330d70 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e740f31efdb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e740f31efdb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 75a6693f2bf5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 75a6693f2bf5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87cde6e5dc84 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87cde6e5dc84 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9e1a619fbdf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9e1a619fbdf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db33efc1576a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db33efc1576a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f5ed887a28e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f5ed887a28e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d7c607195eb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d7c607195eb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8dd3fa943e0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8dd3fa943e0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 632fc2b59127 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 632fc2b59127 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 40632353e752 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 40632353e752 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4542de2cdd7c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4542de2cdd7c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 267c5eb348c0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 267c5eb348c0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05a3effe57f4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05a3effe57f4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8b8a8b6ae0ba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8b8a8b6ae0ba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 517d42a5be73 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 517d42a5be73 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4db75b55daa0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4db75b55daa0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 239aab8e3b8e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 239aab8e3b8e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 541a2417bf58 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 541a2417bf58 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d22023437f2f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d22023437f2f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0a03cd40aa50 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0a03cd40aa50 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ebcdccae0c55 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ebcdccae0c55 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f2eb722d4433 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f2eb722d4433 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae2091b02f50 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae2091b02f50 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48fcb0aa1039 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48fcb0aa1039 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 64a3133ab0f2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 64a3133ab0f2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ace19062a4b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ace19062a4b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e954431f69ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e954431f69ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9ea385ca259 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9ea385ca259 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6cf385d5ab12 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6cf385d5ab12 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7496990b0d2f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7496990b0d2f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a48d823674e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a48d823674e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 42fad6d60c14 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 42fad6d60c14 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f8031ac97262 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f8031ac97262 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aaaad8a618b0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aaaad8a618b0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d3a6e20b704 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d3a6e20b704 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4bf6eb1aa7b7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4bf6eb1aa7b7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9d854bf2b38a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9d854bf2b38a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ca7924e43b13 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ca7924e43b13 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db2218b2ace0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db2218b2ace0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f425fe2e5e78 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f425fe2e5e78 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ae88dd5f394 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ae88dd5f394 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0081c9c4ac92 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0081c9c4ac92 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 226e8a7c158b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 226e8a7c158b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62bb8356f34f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62bb8356f34f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b6fd583aa09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b6fd583aa09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1e544e43128 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1e544e43128 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9604293364e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9604293364e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 295ace22325a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 295ace22325a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 654406ec116e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 654406ec116e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ddd7271fbb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ddd7271fbb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3d6317b211c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3d6317b211c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3c487c43a55 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3c487c43a55 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: daa970146a08 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: daa970146a08 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 018782d2be6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 018782d2be6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 068da4246e1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 068da4246e1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 38ff9644cb28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 38ff9644cb28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 354bbc9ea299 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 354bbc9ea299 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 983e4450f0d1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 983e4450f0d1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 28f1f39ea0e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 28f1f39ea0e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d058647092e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d058647092e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9815d67c9ad2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9815d67c9ad2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4903cdac00ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4903cdac00ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 746c269ca57f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 746c269ca57f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 08621687c353 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 08621687c353 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f341c72d3305 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f341c72d3305 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dd78687617df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dd78687617df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91ad5a8205e5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91ad5a8205e5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 688803c9e58a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 688803c9e58a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68d155660f48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68d155660f48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1a98104a9814 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1a98104a9814 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 51b2cb173001 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 51b2cb173001 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f3be69fc9a8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f3be69fc9a8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27cb775e10a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27cb775e10a0 2024-03-08 10:02:56 proc_lsst.multi:146 [INFO] found job 16914680 in provider checkpoint proc_lsst.multi INFO: found job 16914680 in provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:201 [INFO] cancelling 16914680 on provider checkpoint proc_lsst.multi INFO: cancelling 16914680 on provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:146 [INFO] found job 16914669 in provider checkpoint proc_lsst.multi INFO: found job 16914669 in provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:201 [INFO] cancelling 16914669 on provider checkpoint proc_lsst.multi INFO: cancelling 16914669 on provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:146 [INFO] found job 16914673 in provider checkpoint proc_lsst.multi INFO: found job 16914673 in provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:201 [INFO] cancelling 16914673 on provider checkpoint proc_lsst.multi INFO: cancelling 16914673 on provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:146 [INFO] found job 16914601 in provider astro proc_lsst.multi INFO: found job 16914601 in provider astro 2024-03-08 10:02:56 proc_lsst.multi:201 [INFO] cancelling 16914601 on provider astro proc_lsst.multi INFO: cancelling 16914601 on provider astro 2024-03-08 10:02:56 proc_lsst.multi:146 [INFO] found job 16914668 in provider checkpoint proc_lsst.multi INFO: found job 16914668 in provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:201 [INFO] cancelling 16914668 on provider checkpoint proc_lsst.multi INFO: cancelling 16914668 on provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:146 [INFO] found job 16914667 in provider checkpoint proc_lsst.multi INFO: found job 16914667 in provider checkpoint 2024-03-08 10:02:56 proc_lsst.multi:201 [INFO] cancelling 16914667 on provider checkpoint proc_lsst.multi INFO: cancelling 16914667 on provider checkpoint 2024-03-08 10:02:57 proc_lsst.multi:146 [INFO] found job 16914671 in provider checkpoint proc_lsst.multi INFO: found job 16914671 in provider checkpoint 2024-03-08 10:02:57 proc_lsst.multi:201 [INFO] cancelling 16914671 on provider checkpoint proc_lsst.multi INFO: cancelling 16914671 on provider checkpoint 2024-03-08 10:02:57 proc_lsst.multi:146 [INFO] found job 16914672 in provider checkpoint proc_lsst.multi INFO: found job 16914672 in provider checkpoint 2024-03-08 10:02:57 proc_lsst.multi:201 [INFO] cancelling 16914672 on provider checkpoint proc_lsst.multi INFO: cancelling 16914672 on provider checkpoint 2024-03-08 10:02:57 proc_lsst.multi:146 [INFO] found job 16914630 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914630 in provider escience_rtx6k 2024-03-08 10:02:57 proc_lsst.multi:201 [INFO] cancelling 16914630 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914630 on provider escience_rtx6k 2024-03-08 10:02:57 proc_lsst.multi:146 [INFO] found job 16914656 in provider escience_a40 proc_lsst.multi INFO: found job 16914656 in provider escience_a40 2024-03-08 10:02:57 proc_lsst.multi:201 [INFO] cancelling 16914656 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914656 on provider escience_a40 2024-03-08 10:02:57 proc_lsst.multi:146 [INFO] found job 16914631 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914631 in provider escience_rtx6k 2024-03-08 10:02:57 proc_lsst.multi:201 [INFO] cancelling 16914631 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914631 on provider escience_rtx6k 2024-03-08 10:02:57 proc_lsst.multi:146 [INFO] found job 16914666 in provider checkpoint proc_lsst.multi INFO: found job 16914666 in provider checkpoint 2024-03-08 10:02:57 proc_lsst.multi:201 [INFO] cancelling 16914666 on provider checkpoint proc_lsst.multi INFO: cancelling 16914666 on provider checkpoint 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 16914613 in provider astro proc_lsst.multi INFO: found job 16914613 in provider astro 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 16914613 on provider astro proc_lsst.multi INFO: cancelling 16914613 on provider astro 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 16914660 in provider checkpoint proc_lsst.multi INFO: found job 16914660 in provider checkpoint 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 16914660 on provider checkpoint proc_lsst.multi INFO: cancelling 16914660 on provider checkpoint 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 16914632 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914632 in provider escience_rtx6k 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 16914632 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914632 on provider escience_rtx6k 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 16914685 in provider checkpoint proc_lsst.multi INFO: found job 16914685 in provider checkpoint 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 16914685 on provider checkpoint proc_lsst.multi INFO: cancelling 16914685 on provider checkpoint 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 16914624 in provider astro proc_lsst.multi INFO: found job 16914624 in provider astro 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 16914624 on provider astro proc_lsst.multi INFO: cancelling 16914624 on provider astro 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 30716 in provider local proc_lsst.multi INFO: found job 30716 in provider local 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 30716 on provider local proc_lsst.multi INFO: cancelling 30716 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 30716 2024-03-08 10:02:58 proc_lsst.multi:146 [INFO] found job 16914682 in provider checkpoint proc_lsst.multi INFO: found job 16914682 in provider checkpoint 2024-03-08 10:02:58 proc_lsst.multi:201 [INFO] cancelling 16914682 on provider checkpoint proc_lsst.multi INFO: cancelling 16914682 on provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914657 in provider escience_a40 proc_lsst.multi INFO: found job 16914657 in provider escience_a40 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914657 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914657 on provider escience_a40 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914664 in provider checkpoint proc_lsst.multi INFO: found job 16914664 in provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914664 on provider checkpoint proc_lsst.multi INFO: cancelling 16914664 on provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914681 in provider checkpoint proc_lsst.multi INFO: found job 16914681 in provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914681 on provider checkpoint proc_lsst.multi INFO: cancelling 16914681 on provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914662 in provider checkpoint proc_lsst.multi INFO: found job 16914662 in provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914662 on provider checkpoint proc_lsst.multi INFO: cancelling 16914662 on provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914634 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914634 in provider escience_rtx6k 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914634 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914634 on provider escience_rtx6k 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914665 in provider checkpoint proc_lsst.multi INFO: found job 16914665 in provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914665 on provider checkpoint proc_lsst.multi INFO: cancelling 16914665 on provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914633 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914633 in provider escience_rtx6k 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914633 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914633 on provider escience_rtx6k 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914678 in provider checkpoint proc_lsst.multi INFO: found job 16914678 in provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914678 on provider checkpoint proc_lsst.multi INFO: cancelling 16914678 on provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:146 [INFO] found job 16914677 in provider checkpoint proc_lsst.multi INFO: found job 16914677 in provider checkpoint 2024-03-08 10:02:59 proc_lsst.multi:201 [INFO] cancelling 16914677 on provider checkpoint proc_lsst.multi INFO: cancelling 16914677 on provider checkpoint 2024-03-08 10:03:00 proc_lsst.multi:146 [INFO] found job 16914647 in provider escience_a40 proc_lsst.multi INFO: found job 16914647 in provider escience_a40 2024-03-08 10:03:00 proc_lsst.multi:201 [INFO] cancelling 16914647 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914647 on provider escience_a40 2024-03-08 10:03:00 proc_lsst.multi:146 [INFO] found job 16914674 in provider checkpoint proc_lsst.multi INFO: found job 16914674 in provider checkpoint 2024-03-08 10:03:00 proc_lsst.multi:201 [INFO] cancelling 16914674 on provider checkpoint proc_lsst.multi INFO: cancelling 16914674 on provider checkpoint 2024-03-08 10:03:00 proc_lsst.multi:146 [INFO] found job 16914675 in provider checkpoint proc_lsst.multi INFO: found job 16914675 in provider checkpoint 2024-03-08 10:03:00 proc_lsst.multi:201 [INFO] cancelling 16914675 on provider checkpoint proc_lsst.multi INFO: cancelling 16914675 on provider checkpoint 2024-03-08 10:03:00 proc_lsst.multi:146 [INFO] found job 16914658 in provider escience_a40 proc_lsst.multi INFO: found job 16914658 in provider escience_a40 2024-03-08 10:03:00 proc_lsst.multi:201 [INFO] cancelling 16914658 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914658 on provider escience_a40 2024-03-08 10:03:00 proc_lsst.multi:146 [INFO] found job 16914604 in provider astro proc_lsst.multi INFO: found job 16914604 in provider astro 2024-03-08 10:03:00 proc_lsst.multi:201 [INFO] cancelling 16914604 on provider astro proc_lsst.multi INFO: cancelling 16914604 on provider astro 2024-03-08 10:03:00 proc_lsst.multi:146 [INFO] found job 16914659 in provider checkpoint proc_lsst.multi INFO: found job 16914659 in provider checkpoint 2024-03-08 10:03:00 proc_lsst.multi:201 [INFO] cancelling 16914659 on provider checkpoint proc_lsst.multi INFO: cancelling 16914659 on provider checkpoint 2024-03-08 10:03:01 proc_lsst.multi:146 [INFO] found job 16914638 in provider escience_a40 proc_lsst.multi INFO: found job 16914638 in provider escience_a40 2024-03-08 10:03:01 proc_lsst.multi:201 [INFO] cancelling 16914638 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914638 on provider escience_a40 2024-03-08 10:03:01 proc_lsst.multi:146 [INFO] found job 16914616 in provider astro proc_lsst.multi INFO: found job 16914616 in provider astro 2024-03-08 10:03:01 proc_lsst.multi:201 [INFO] cancelling 16914616 on provider astro proc_lsst.multi INFO: cancelling 16914616 on provider astro 2024-03-08 10:03:01 proc_lsst.multi:146 [INFO] found job 16914600 in provider astro proc_lsst.multi INFO: found job 16914600 in provider astro 2024-03-08 10:03:01 proc_lsst.multi:201 [INFO] cancelling 16914600 on provider astro proc_lsst.multi INFO: cancelling 16914600 on provider astro 2024-03-08 10:03:01 proc_lsst.multi:146 [INFO] found job 16914651 in provider escience_a40 proc_lsst.multi INFO: found job 16914651 in provider escience_a40 2024-03-08 10:03:01 proc_lsst.multi:201 [INFO] cancelling 16914651 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914651 on provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:146 [INFO] found job 16914639 in provider escience_a40 proc_lsst.multi INFO: found job 16914639 in provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:201 [INFO] cancelling 16914639 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914639 on provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:146 [INFO] found job 16914650 in provider escience_a40 proc_lsst.multi INFO: found job 16914650 in provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:201 [INFO] cancelling 16914650 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914650 on provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:146 [INFO] found job 16914641 in provider escience_a40 proc_lsst.multi INFO: found job 16914641 in provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:201 [INFO] cancelling 16914641 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914641 on provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:146 [INFO] found job 16914654 in provider escience_a40 proc_lsst.multi INFO: found job 16914654 in provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:201 [INFO] cancelling 16914654 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914654 on provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:146 [INFO] found job 16914643 in provider escience_a40 proc_lsst.multi INFO: found job 16914643 in provider escience_a40 2024-03-08 10:03:02 proc_lsst.multi:201 [INFO] cancelling 16914643 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914643 on provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:146 [INFO] found job 16914625 in provider astro proc_lsst.multi INFO: found job 16914625 in provider astro 2024-03-08 10:03:03 proc_lsst.multi:201 [INFO] cancelling 16914625 on provider astro proc_lsst.multi INFO: cancelling 16914625 on provider astro 2024-03-08 10:03:03 proc_lsst.multi:146 [INFO] found job 16914642 in provider escience_a40 proc_lsst.multi INFO: found job 16914642 in provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:201 [INFO] cancelling 16914642 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914642 on provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:146 [INFO] found job 16914649 in provider escience_a40 proc_lsst.multi INFO: found job 16914649 in provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:201 [INFO] cancelling 16914649 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914649 on provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:146 [INFO] found job 16914652 in provider escience_a40 proc_lsst.multi INFO: found job 16914652 in provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:201 [INFO] cancelling 16914652 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914652 on provider escience_a40 2024-03-08 10:03:03 proc_lsst.multi:146 [INFO] found job 16914670 in provider checkpoint proc_lsst.multi INFO: found job 16914670 in provider checkpoint 2024-03-08 10:03:03 proc_lsst.multi:201 [INFO] cancelling 16914670 on provider checkpoint proc_lsst.multi INFO: cancelling 16914670 on provider checkpoint 2024-03-08 10:03:04 proc_lsst.multi:146 [INFO] found job 16914648 in provider escience_a40 proc_lsst.multi INFO: found job 16914648 in provider escience_a40 2024-03-08 10:03:04 proc_lsst.multi:201 [INFO] cancelling 16914648 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914648 on provider escience_a40 2024-03-08 10:03:04 proc_lsst.multi:146 [INFO] found job 16914655 in provider escience_a40 proc_lsst.multi INFO: found job 16914655 in provider escience_a40 2024-03-08 10:03:04 proc_lsst.multi:201 [INFO] cancelling 16914655 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914655 on provider escience_a40 2024-03-08 10:03:04 proc_lsst.multi:146 [INFO] found job 16914640 in provider escience_a40 proc_lsst.multi INFO: found job 16914640 in provider escience_a40 2024-03-08 10:03:04 proc_lsst.multi:201 [INFO] cancelling 16914640 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914640 on provider escience_a40 2024-03-08 10:03:04 proc_lsst.multi:146 [INFO] found job 16914684 in provider checkpoint proc_lsst.multi INFO: found job 16914684 in provider checkpoint 2024-03-08 10:03:04 proc_lsst.multi:201 [INFO] cancelling 16914684 on provider checkpoint proc_lsst.multi INFO: cancelling 16914684 on provider checkpoint 2024-03-08 10:03:04 proc_lsst.multi:146 [INFO] found job 16914622 in provider astro proc_lsst.multi INFO: found job 16914622 in provider astro 2024-03-08 10:03:04 proc_lsst.multi:201 [INFO] cancelling 16914622 on provider astro proc_lsst.multi INFO: cancelling 16914622 on provider astro 2024-03-08 10:03:04 proc_lsst.multi:146 [INFO] found job 16914611 in provider astro proc_lsst.multi INFO: found job 16914611 in provider astro 2024-03-08 10:03:04 proc_lsst.multi:201 [INFO] cancelling 16914611 on provider astro proc_lsst.multi INFO: cancelling 16914611 on provider astro 2024-03-08 10:03:05 proc_lsst.multi:146 [INFO] found job 16914607 in provider astro proc_lsst.multi INFO: found job 16914607 in provider astro 2024-03-08 10:03:05 proc_lsst.multi:201 [INFO] cancelling 16914607 on provider astro proc_lsst.multi INFO: cancelling 16914607 on provider astro 2024-03-08 10:03:05 proc_lsst.multi:146 [INFO] found job 16914602 in provider astro proc_lsst.multi INFO: found job 16914602 in provider astro 2024-03-08 10:03:05 proc_lsst.multi:201 [INFO] cancelling 16914602 on provider astro proc_lsst.multi INFO: cancelling 16914602 on provider astro 2024-03-08 10:03:05 proc_lsst.multi:146 [INFO] found job 16914637 in provider escience_a40 proc_lsst.multi INFO: found job 16914637 in provider escience_a40 2024-03-08 10:03:05 proc_lsst.multi:201 [INFO] cancelling 16914637 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914637 on provider escience_a40 2024-03-08 10:03:05 proc_lsst.multi:146 [INFO] found job 16914603 in provider astro proc_lsst.multi INFO: found job 16914603 in provider astro 2024-03-08 10:03:05 proc_lsst.multi:201 [INFO] cancelling 16914603 on provider astro proc_lsst.multi INFO: cancelling 16914603 on provider astro 2024-03-08 10:03:05 proc_lsst.multi:146 [INFO] found job 16914663 in provider checkpoint proc_lsst.multi INFO: found job 16914663 in provider checkpoint 2024-03-08 10:03:05 proc_lsst.multi:201 [INFO] cancelling 16914663 on provider checkpoint proc_lsst.multi INFO: cancelling 16914663 on provider checkpoint 2024-03-08 10:03:06 proc_lsst.multi:146 [INFO] found job 16914618 in provider astro proc_lsst.multi INFO: found job 16914618 in provider astro 2024-03-08 10:03:06 proc_lsst.multi:201 [INFO] cancelling 16914618 on provider astro proc_lsst.multi INFO: cancelling 16914618 on provider astro 2024-03-08 10:03:06 proc_lsst.multi:146 [INFO] found job 16914612 in provider astro proc_lsst.multi INFO: found job 16914612 in provider astro 2024-03-08 10:03:06 proc_lsst.multi:201 [INFO] cancelling 16914612 on provider astro proc_lsst.multi INFO: cancelling 16914612 on provider astro 2024-03-08 10:03:06 proc_lsst.multi:146 [INFO] found job 16914679 in provider checkpoint proc_lsst.multi INFO: found job 16914679 in provider checkpoint 2024-03-08 10:03:06 proc_lsst.multi:201 [INFO] cancelling 16914679 on provider checkpoint proc_lsst.multi INFO: cancelling 16914679 on provider checkpoint 2024-03-08 10:03:06 proc_lsst.multi:146 [INFO] found job 16914619 in provider astro proc_lsst.multi INFO: found job 16914619 in provider astro 2024-03-08 10:03:06 proc_lsst.multi:201 [INFO] cancelling 16914619 on provider astro proc_lsst.multi INFO: cancelling 16914619 on provider astro 2024-03-08 10:03:06 proc_lsst.multi:146 [INFO] found job 16914608 in provider astro proc_lsst.multi INFO: found job 16914608 in provider astro 2024-03-08 10:03:06 proc_lsst.multi:201 [INFO] cancelling 16914608 on provider astro proc_lsst.multi INFO: cancelling 16914608 on provider astro 2024-03-08 10:03:07 proc_lsst.multi:146 [INFO] found job 16914615 in provider astro proc_lsst.multi INFO: found job 16914615 in provider astro 2024-03-08 10:03:07 proc_lsst.multi:201 [INFO] cancelling 16914615 on provider astro proc_lsst.multi INFO: cancelling 16914615 on provider astro 2024-03-08 10:03:07 proc_lsst.multi:146 [INFO] found job 16914609 in provider astro proc_lsst.multi INFO: found job 16914609 in provider astro 2024-03-08 10:03:07 proc_lsst.multi:201 [INFO] cancelling 16914609 on provider astro proc_lsst.multi INFO: cancelling 16914609 on provider astro 2024-03-08 10:03:07 proc_lsst.multi:146 [INFO] found job 16914623 in provider astro proc_lsst.multi INFO: found job 16914623 in provider astro 2024-03-08 10:03:07 proc_lsst.multi:201 [INFO] cancelling 16914623 on provider astro proc_lsst.multi INFO: cancelling 16914623 on provider astro 2024-03-08 10:03:07 proc_lsst.multi:146 [INFO] found job 16914636 in provider escience_a40 proc_lsst.multi INFO: found job 16914636 in provider escience_a40 2024-03-08 10:03:07 proc_lsst.multi:201 [INFO] cancelling 16914636 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914636 on provider escience_a40 2024-03-08 10:03:07 proc_lsst.multi:146 [INFO] found job 16914661 in provider checkpoint proc_lsst.multi INFO: found job 16914661 in provider checkpoint 2024-03-08 10:03:07 proc_lsst.multi:201 [INFO] cancelling 16914661 on provider checkpoint proc_lsst.multi INFO: cancelling 16914661 on provider checkpoint 2024-03-08 10:03:07 proc_lsst.multi:146 [INFO] found job 16914621 in provider astro proc_lsst.multi INFO: found job 16914621 in provider astro 2024-03-08 10:03:07 proc_lsst.multi:201 [INFO] cancelling 16914621 on provider astro proc_lsst.multi INFO: cancelling 16914621 on provider astro 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914635 in provider escience_a40 proc_lsst.multi INFO: found job 16914635 in provider escience_a40 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914635 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914635 on provider escience_a40 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914617 in provider astro proc_lsst.multi INFO: found job 16914617 in provider astro 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914617 on provider astro proc_lsst.multi INFO: cancelling 16914617 on provider astro 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914605 in provider astro proc_lsst.multi INFO: found job 16914605 in provider astro 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914605 on provider astro proc_lsst.multi INFO: cancelling 16914605 on provider astro 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914620 in provider astro proc_lsst.multi INFO: found job 16914620 in provider astro 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914620 on provider astro proc_lsst.multi INFO: cancelling 16914620 on provider astro 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914627 in provider astro proc_lsst.multi INFO: found job 16914627 in provider astro 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914627 on provider astro proc_lsst.multi INFO: cancelling 16914627 on provider astro 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914628 in provider astro proc_lsst.multi INFO: found job 16914628 in provider astro 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914628 on provider astro proc_lsst.multi INFO: cancelling 16914628 on provider astro 2024-03-08 10:03:08 proc_lsst.multi:146 [INFO] found job 16914610 in provider astro proc_lsst.multi INFO: found job 16914610 in provider astro 2024-03-08 10:03:08 proc_lsst.multi:201 [INFO] cancelling 16914610 on provider astro proc_lsst.multi INFO: cancelling 16914610 on provider astro 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914606 in provider astro proc_lsst.multi INFO: found job 16914606 in provider astro 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914606 on provider astro proc_lsst.multi INFO: cancelling 16914606 on provider astro 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914614 in provider astro proc_lsst.multi INFO: found job 16914614 in provider astro 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914614 on provider astro proc_lsst.multi INFO: cancelling 16914614 on provider astro 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914683 in provider checkpoint proc_lsst.multi INFO: found job 16914683 in provider checkpoint 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914683 on provider checkpoint proc_lsst.multi INFO: cancelling 16914683 on provider checkpoint 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914626 in provider astro proc_lsst.multi INFO: found job 16914626 in provider astro 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914626 on provider astro proc_lsst.multi INFO: cancelling 16914626 on provider astro 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914629 in provider astro proc_lsst.multi INFO: found job 16914629 in provider astro 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914629 on provider astro proc_lsst.multi INFO: cancelling 16914629 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-08 10:03:09 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-08 10:03:09 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16914659', '16914660', '16914661', '16914662', '16914663', '16914664', '16914665', '16914666', '16914667', '16914668', '16914669', '16914670', '16914671', '16914672', '16914673', '16914674', '16914675', '16914677', '16914678', '16914679', '16914680', '16914681', '16914682', '16914683', '16914684', '16914685', '16914686', '16914687', '16914688', '16914689', '16914690', '16914691', '16914701', '16914702', '16914703', '16914704', '16914705', '16914706', '16914707', '16914708', '16914709', '16914710', '16914711', '16914712', '16914713', '16914714', '16914715', '16914716', '16914717', '16914718', '16914719', '16914720', '16914721', '16914722', '16914723', '16914724', '16914725', '16914726', '16914727', '16914728', '16914729', '16914730', '16914731', '16914732', '16914733', '16914734', '16914735', '16914736', '16914737', '16914738', '16914739', '16914740', '16914741', '16914742', '16914743', '16914744', '16914745', '16914746', '16914756', '16914757', '16914758', '16914759', '16914760', '16914761', '16914762', '16914763', '16914764', '16914765', '16914766', '16914767', '16914768', '16914769', '16914770', '16914771', '16914772', '16914773', '16914774', '16914775', '16914776', '16914777', '16914778', '16914779', '16914780', '16914781', '16914782', '16914783', '16914784', '16914785', '16914786', '16914787', '16914788', '16914789', '16914790', '16914791', '16914792', '16914793', '16914794', '16914800', '16914801', '16914802', '16914803', '16914804', '16914805', '16914806', '16914807', '16914808', '16914809', '16914810', '16914811', '16914812', '16914813', '16914814', '16914815', '16914816', '16914817', '16914818', '16914819', '16914820', '16914821', '16914822', '16914823', '16914824', '16914825', '16914826', '16914827', '16914828', '16914829', '16914830', '16914831', '16914832', '16914833', '16914834', '16914835', '16914836', '16914837', '16914838', '16914839', '16914840', '16914841', '16914842', '16914843', '16914844', '16914845', '16914846', '16914847', '16914848', '16914849', '16914850', '16914851', '16914852', '16914853', '16914854', '16914855', '16914856', '16914857', '16914858', '16914600', '16914601', '16914602', '16914603', '16914604', '16914605', '16914606', '16914607', '16914608', '16914609', '16914610', '16914611', '16914612', '16914613', '16914614', '16914615', '16914616', '16914617', '16914618', '16914619', '16914620', '16914621', '16914622', '16914623', '16914624', '16914625', '16914626', '16914627', '16914628', '16914629', '16914635', '16914636', '16914637', '16914638', '16914639', '16914640', '16914641', '16914642', '16914643', '16914647', '16914648', '16914649', '16914650', '16914651', '16914652', '16914654', '16914655', '16914656', '16914657', '16914658', '16914630', '16914631', '16914632', '16914633', '16914634', '30716'] proc_lsst.multi INFO: new jobs since last cancel ['16914659', '16914660', '16914661', '16914662', '16914663', '16914664', '16914665', '16914666', '16914667', '16914668', '16914669', '16914670', '16914671', '16914672', '16914673', '16914674', '16914675', '16914677', '16914678', '16914679', '16914680', '16914681', '16914682', '16914683', '16914684', '16914685', '16914686', '16914687', '16914688', '16914689', '16914690', '16914691', '16914701', '16914702', '16914703', '16914704', '16914705', '16914706', '16914707', '16914708', '16914709', '16914710', '16914711', '16914712', '16914713', '16914714', '16914715', '16914716', '16914717', '16914718', '16914719', '16914720', '16914721', '16914722', '16914723', '16914724', '16914725', '16914726', '16914727', '16914728', '16914729', '16914730', '16914731', '16914732', '16914733', '16914734', '16914735', '16914736', '16914737', '16914738', '16914739', '16914740', '16914741', '16914742', '16914743', '16914744', '16914745', '16914746', '16914756', '16914757', '16914758', '16914759', '16914760', '16914761', '16914762', '16914763', '16914764', '16914765', '16914766', '16914767', '16914768', '16914769', '16914770', '16914771', '16914772', '16914773', '16914774', '16914775', '16914776', '16914777', '16914778', '16914779', '16914780', '16914781', '16914782', '16914783', '16914784', '16914785', '16914786', '16914787', '16914788', '16914789', '16914790', '16914791', '16914792', '16914793', '16914794', '16914800', '16914801', '16914802', '16914803', '16914804', '16914805', '16914806', '16914807', '16914808', '16914809', '16914810', '16914811', '16914812', '16914813', '16914814', '16914815', '16914816', '16914817', '16914818', '16914819', '16914820', '16914821', '16914822', '16914823', '16914824', '16914825', '16914826', '16914827', '16914828', '16914829', '16914830', '16914831', '16914832', '16914833', '16914834', '16914835', '16914836', '16914837', '16914838', '16914839', '16914840', '16914841', '16914842', '16914843', '16914844', '16914845', '16914846', '16914847', '16914848', '16914849', '16914850', '16914851', '16914852', '16914853', '16914854', '16914855', '16914856', '16914857', '16914858', '16914600', '16914601', '16914602', '16914603', '16914604', '16914605', '16914606', '16914607', '16914608', '16914609', '16914610', '16914611', '16914612', '16914613', '16914614', '16914615', '16914616', '16914617', '16914618', '16914619', '16914620', '16914621', '16914622', '16914623', '16914624', '16914625', '16914626', '16914627', '16914628', '16914629', '16914635', '16914636', '16914637', '16914638', '16914639', '16914640', '16914641', '16914642', '16914643', '16914647', '16914648', '16914649', '16914650', '16914651', '16914652', '16914654', '16914655', '16914656', '16914657', '16914658', '16914630', '16914631', '16914632', '16914633', '16914634', '30716'] 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914659 in provider checkpoint proc_lsst.multi INFO: found job 16914659 in provider checkpoint 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914659 on provider checkpoint proc_lsst.multi INFO: cancelling 16914659 on provider checkpoint 2024-03-08 10:03:09 proc_lsst.multi:146 [INFO] found job 16914660 in provider checkpoint proc_lsst.multi INFO: found job 16914660 in provider checkpoint 2024-03-08 10:03:09 proc_lsst.multi:201 [INFO] cancelling 16914660 on provider checkpoint proc_lsst.multi INFO: cancelling 16914660 on provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:146 [INFO] found job 16914661 in provider checkpoint proc_lsst.multi INFO: found job 16914661 in provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:201 [INFO] cancelling 16914661 on provider checkpoint proc_lsst.multi INFO: cancelling 16914661 on provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:146 [INFO] found job 16914662 in provider checkpoint proc_lsst.multi INFO: found job 16914662 in provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:201 [INFO] cancelling 16914662 on provider checkpoint proc_lsst.multi INFO: cancelling 16914662 on provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:146 [INFO] found job 16914663 in provider checkpoint proc_lsst.multi INFO: found job 16914663 in provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:201 [INFO] cancelling 16914663 on provider checkpoint proc_lsst.multi INFO: cancelling 16914663 on provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:146 [INFO] found job 16914664 in provider checkpoint proc_lsst.multi INFO: found job 16914664 in provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:201 [INFO] cancelling 16914664 on provider checkpoint proc_lsst.multi INFO: cancelling 16914664 on provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:146 [INFO] found job 16914665 in provider checkpoint proc_lsst.multi INFO: found job 16914665 in provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:201 [INFO] cancelling 16914665 on provider checkpoint proc_lsst.multi INFO: cancelling 16914665 on provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:146 [INFO] found job 16914666 in provider checkpoint proc_lsst.multi INFO: found job 16914666 in provider checkpoint 2024-03-08 10:03:10 proc_lsst.multi:201 [INFO] cancelling 16914666 on provider checkpoint proc_lsst.multi INFO: cancelling 16914666 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914667 in provider checkpoint proc_lsst.multi INFO: found job 16914667 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914667 on provider checkpoint proc_lsst.multi INFO: cancelling 16914667 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914668 in provider checkpoint proc_lsst.multi INFO: found job 16914668 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914668 on provider checkpoint proc_lsst.multi INFO: cancelling 16914668 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914669 in provider checkpoint proc_lsst.multi INFO: found job 16914669 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914669 on provider checkpoint proc_lsst.multi INFO: cancelling 16914669 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914670 in provider checkpoint proc_lsst.multi INFO: found job 16914670 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914670 on provider checkpoint proc_lsst.multi INFO: cancelling 16914670 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914671 in provider checkpoint proc_lsst.multi INFO: found job 16914671 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914671 on provider checkpoint proc_lsst.multi INFO: cancelling 16914671 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914672 in provider checkpoint proc_lsst.multi INFO: found job 16914672 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914672 on provider checkpoint proc_lsst.multi INFO: cancelling 16914672 on provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:146 [INFO] found job 16914673 in provider checkpoint proc_lsst.multi INFO: found job 16914673 in provider checkpoint 2024-03-08 10:03:11 proc_lsst.multi:201 [INFO] cancelling 16914673 on provider checkpoint proc_lsst.multi INFO: cancelling 16914673 on provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:146 [INFO] found job 16914674 in provider checkpoint proc_lsst.multi INFO: found job 16914674 in provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:201 [INFO] cancelling 16914674 on provider checkpoint proc_lsst.multi INFO: cancelling 16914674 on provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:146 [INFO] found job 16914675 in provider checkpoint proc_lsst.multi INFO: found job 16914675 in provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:201 [INFO] cancelling 16914675 on provider checkpoint proc_lsst.multi INFO: cancelling 16914675 on provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:146 [INFO] found job 16914677 in provider checkpoint proc_lsst.multi INFO: found job 16914677 in provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:201 [INFO] cancelling 16914677 on provider checkpoint proc_lsst.multi INFO: cancelling 16914677 on provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:146 [INFO] found job 16914678 in provider checkpoint proc_lsst.multi INFO: found job 16914678 in provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:201 [INFO] cancelling 16914678 on provider checkpoint proc_lsst.multi INFO: cancelling 16914678 on provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:146 [INFO] found job 16914679 in provider checkpoint proc_lsst.multi INFO: found job 16914679 in provider checkpoint 2024-03-08 10:03:12 proc_lsst.multi:201 [INFO] cancelling 16914679 on provider checkpoint proc_lsst.multi INFO: cancelling 16914679 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914680 in provider checkpoint proc_lsst.multi INFO: found job 16914680 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914680 on provider checkpoint proc_lsst.multi INFO: cancelling 16914680 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914681 in provider checkpoint proc_lsst.multi INFO: found job 16914681 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914681 on provider checkpoint proc_lsst.multi INFO: cancelling 16914681 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914682 in provider checkpoint proc_lsst.multi INFO: found job 16914682 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914682 on provider checkpoint proc_lsst.multi INFO: cancelling 16914682 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914683 in provider checkpoint proc_lsst.multi INFO: found job 16914683 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914683 on provider checkpoint proc_lsst.multi INFO: cancelling 16914683 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914684 in provider checkpoint proc_lsst.multi INFO: found job 16914684 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914684 on provider checkpoint proc_lsst.multi INFO: cancelling 16914684 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914685 in provider checkpoint proc_lsst.multi INFO: found job 16914685 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914685 on provider checkpoint proc_lsst.multi INFO: cancelling 16914685 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914686 in provider checkpoint proc_lsst.multi INFO: found job 16914686 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914686 on provider checkpoint proc_lsst.multi INFO: cancelling 16914686 on provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:146 [INFO] found job 16914687 in provider checkpoint proc_lsst.multi INFO: found job 16914687 in provider checkpoint 2024-03-08 10:03:13 proc_lsst.multi:201 [INFO] cancelling 16914687 on provider checkpoint proc_lsst.multi INFO: cancelling 16914687 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914688 in provider checkpoint proc_lsst.multi INFO: found job 16914688 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914688 on provider checkpoint proc_lsst.multi INFO: cancelling 16914688 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914689 in provider checkpoint proc_lsst.multi INFO: found job 16914689 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914689 on provider checkpoint proc_lsst.multi INFO: cancelling 16914689 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914690 in provider checkpoint proc_lsst.multi INFO: found job 16914690 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914690 on provider checkpoint proc_lsst.multi INFO: cancelling 16914690 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914691 in provider checkpoint proc_lsst.multi INFO: found job 16914691 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914691 on provider checkpoint proc_lsst.multi INFO: cancelling 16914691 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914701 in provider checkpoint proc_lsst.multi INFO: found job 16914701 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914701 on provider checkpoint proc_lsst.multi INFO: cancelling 16914701 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914702 in provider checkpoint proc_lsst.multi INFO: found job 16914702 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914702 on provider checkpoint proc_lsst.multi INFO: cancelling 16914702 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914703 in provider checkpoint proc_lsst.multi INFO: found job 16914703 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914703 on provider checkpoint proc_lsst.multi INFO: cancelling 16914703 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914704 in provider checkpoint proc_lsst.multi INFO: found job 16914704 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914704 on provider checkpoint proc_lsst.multi INFO: cancelling 16914704 on provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:146 [INFO] found job 16914705 in provider checkpoint proc_lsst.multi INFO: found job 16914705 in provider checkpoint 2024-03-08 10:03:14 proc_lsst.multi:201 [INFO] cancelling 16914705 on provider checkpoint proc_lsst.multi INFO: cancelling 16914705 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914706 in provider checkpoint proc_lsst.multi INFO: found job 16914706 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914706 on provider checkpoint proc_lsst.multi INFO: cancelling 16914706 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914707 in provider checkpoint proc_lsst.multi INFO: found job 16914707 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914707 on provider checkpoint proc_lsst.multi INFO: cancelling 16914707 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914708 in provider checkpoint proc_lsst.multi INFO: found job 16914708 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914708 on provider checkpoint proc_lsst.multi INFO: cancelling 16914708 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914709 in provider checkpoint proc_lsst.multi INFO: found job 16914709 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914709 on provider checkpoint proc_lsst.multi INFO: cancelling 16914709 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914710 in provider checkpoint proc_lsst.multi INFO: found job 16914710 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914710 on provider checkpoint proc_lsst.multi INFO: cancelling 16914710 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914711 in provider checkpoint proc_lsst.multi INFO: found job 16914711 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914711 on provider checkpoint proc_lsst.multi INFO: cancelling 16914711 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914712 in provider checkpoint proc_lsst.multi INFO: found job 16914712 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914712 on provider checkpoint proc_lsst.multi INFO: cancelling 16914712 on provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:146 [INFO] found job 16914713 in provider checkpoint proc_lsst.multi INFO: found job 16914713 in provider checkpoint 2024-03-08 10:03:15 proc_lsst.multi:201 [INFO] cancelling 16914713 on provider checkpoint proc_lsst.multi INFO: cancelling 16914713 on provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:146 [INFO] found job 16914714 in provider checkpoint proc_lsst.multi INFO: found job 16914714 in provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:201 [INFO] cancelling 16914714 on provider checkpoint proc_lsst.multi INFO: cancelling 16914714 on provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:146 [INFO] found job 16914715 in provider checkpoint proc_lsst.multi INFO: found job 16914715 in provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:201 [INFO] cancelling 16914715 on provider checkpoint proc_lsst.multi INFO: cancelling 16914715 on provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:146 [INFO] found job 16914716 in provider checkpoint proc_lsst.multi INFO: found job 16914716 in provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:201 [INFO] cancelling 16914716 on provider checkpoint proc_lsst.multi INFO: cancelling 16914716 on provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:146 [INFO] found job 16914717 in provider checkpoint proc_lsst.multi INFO: found job 16914717 in provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:201 [INFO] cancelling 16914717 on provider checkpoint proc_lsst.multi INFO: cancelling 16914717 on provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:146 [INFO] found job 16914718 in provider checkpoint proc_lsst.multi INFO: found job 16914718 in provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:201 [INFO] cancelling 16914718 on provider checkpoint proc_lsst.multi INFO: cancelling 16914718 on provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:146 [INFO] found job 16914719 in provider checkpoint proc_lsst.multi INFO: found job 16914719 in provider checkpoint 2024-03-08 10:03:16 proc_lsst.multi:201 [INFO] cancelling 16914719 on provider checkpoint proc_lsst.multi INFO: cancelling 16914719 on provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:146 [INFO] found job 16914720 in provider checkpoint proc_lsst.multi INFO: found job 16914720 in provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:201 [INFO] cancelling 16914720 on provider checkpoint proc_lsst.multi INFO: cancelling 16914720 on provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:146 [INFO] found job 16914721 in provider checkpoint proc_lsst.multi INFO: found job 16914721 in provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:201 [INFO] cancelling 16914721 on provider checkpoint proc_lsst.multi INFO: cancelling 16914721 on provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:146 [INFO] found job 16914722 in provider checkpoint proc_lsst.multi INFO: found job 16914722 in provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:201 [INFO] cancelling 16914722 on provider checkpoint proc_lsst.multi INFO: cancelling 16914722 on provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:146 [INFO] found job 16914723 in provider checkpoint proc_lsst.multi INFO: found job 16914723 in provider checkpoint 2024-03-08 10:03:18 proc_lsst.multi:201 [INFO] cancelling 16914723 on provider checkpoint proc_lsst.multi INFO: cancelling 16914723 on provider checkpoint 2024-03-08 10:03:19 proc_lsst.multi:146 [INFO] found job 16914724 in provider checkpoint proc_lsst.multi INFO: found job 16914724 in provider checkpoint 2024-03-08 10:03:19 proc_lsst.multi:201 [INFO] cancelling 16914724 on provider checkpoint proc_lsst.multi INFO: cancelling 16914724 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914725 in provider checkpoint proc_lsst.multi INFO: found job 16914725 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914725 on provider checkpoint proc_lsst.multi INFO: cancelling 16914725 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914726 in provider checkpoint proc_lsst.multi INFO: found job 16914726 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914726 on provider checkpoint proc_lsst.multi INFO: cancelling 16914726 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914727 in provider checkpoint proc_lsst.multi INFO: found job 16914727 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914727 on provider checkpoint proc_lsst.multi INFO: cancelling 16914727 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914728 in provider checkpoint proc_lsst.multi INFO: found job 16914728 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914728 on provider checkpoint proc_lsst.multi INFO: cancelling 16914728 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914729 in provider checkpoint proc_lsst.multi INFO: found job 16914729 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914729 on provider checkpoint proc_lsst.multi INFO: cancelling 16914729 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914730 in provider checkpoint proc_lsst.multi INFO: found job 16914730 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914730 on provider checkpoint proc_lsst.multi INFO: cancelling 16914730 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914731 in provider checkpoint proc_lsst.multi INFO: found job 16914731 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914731 on provider checkpoint proc_lsst.multi INFO: cancelling 16914731 on provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:146 [INFO] found job 16914732 in provider checkpoint proc_lsst.multi INFO: found job 16914732 in provider checkpoint 2024-03-08 10:03:20 proc_lsst.multi:201 [INFO] cancelling 16914732 on provider checkpoint proc_lsst.multi INFO: cancelling 16914732 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914733 in provider checkpoint proc_lsst.multi INFO: found job 16914733 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914733 on provider checkpoint proc_lsst.multi INFO: cancelling 16914733 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914734 in provider checkpoint proc_lsst.multi INFO: found job 16914734 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914734 on provider checkpoint proc_lsst.multi INFO: cancelling 16914734 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914735 in provider checkpoint proc_lsst.multi INFO: found job 16914735 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914735 on provider checkpoint proc_lsst.multi INFO: cancelling 16914735 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914736 in provider checkpoint proc_lsst.multi INFO: found job 16914736 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914736 on provider checkpoint proc_lsst.multi INFO: cancelling 16914736 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914737 in provider checkpoint proc_lsst.multi INFO: found job 16914737 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914737 on provider checkpoint proc_lsst.multi INFO: cancelling 16914737 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914738 in provider checkpoint proc_lsst.multi INFO: found job 16914738 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914738 on provider checkpoint proc_lsst.multi INFO: cancelling 16914738 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914739 in provider checkpoint proc_lsst.multi INFO: found job 16914739 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914739 on provider checkpoint proc_lsst.multi INFO: cancelling 16914739 on provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:146 [INFO] found job 16914740 in provider checkpoint proc_lsst.multi INFO: found job 16914740 in provider checkpoint 2024-03-08 10:03:21 proc_lsst.multi:201 [INFO] cancelling 16914740 on provider checkpoint proc_lsst.multi INFO: cancelling 16914740 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914741 in provider checkpoint proc_lsst.multi INFO: found job 16914741 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914741 on provider checkpoint proc_lsst.multi INFO: cancelling 16914741 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914742 in provider checkpoint proc_lsst.multi INFO: found job 16914742 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914742 on provider checkpoint proc_lsst.multi INFO: cancelling 16914742 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914743 in provider checkpoint proc_lsst.multi INFO: found job 16914743 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914743 on provider checkpoint proc_lsst.multi INFO: cancelling 16914743 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914744 in provider checkpoint proc_lsst.multi INFO: found job 16914744 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914744 on provider checkpoint proc_lsst.multi INFO: cancelling 16914744 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914745 in provider checkpoint proc_lsst.multi INFO: found job 16914745 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914745 on provider checkpoint proc_lsst.multi INFO: cancelling 16914745 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914746 in provider checkpoint proc_lsst.multi INFO: found job 16914746 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914746 on provider checkpoint proc_lsst.multi INFO: cancelling 16914746 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914756 in provider checkpoint proc_lsst.multi INFO: found job 16914756 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914756 on provider checkpoint proc_lsst.multi INFO: cancelling 16914756 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914757 in provider checkpoint proc_lsst.multi INFO: found job 16914757 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914757 on provider checkpoint proc_lsst.multi INFO: cancelling 16914757 on provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:146 [INFO] found job 16914758 in provider checkpoint proc_lsst.multi INFO: found job 16914758 in provider checkpoint 2024-03-08 10:03:22 proc_lsst.multi:201 [INFO] cancelling 16914758 on provider checkpoint proc_lsst.multi INFO: cancelling 16914758 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914759 in provider checkpoint proc_lsst.multi INFO: found job 16914759 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914759 on provider checkpoint proc_lsst.multi INFO: cancelling 16914759 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914760 in provider checkpoint proc_lsst.multi INFO: found job 16914760 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914760 on provider checkpoint proc_lsst.multi INFO: cancelling 16914760 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914761 in provider checkpoint proc_lsst.multi INFO: found job 16914761 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914761 on provider checkpoint proc_lsst.multi INFO: cancelling 16914761 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914762 in provider checkpoint proc_lsst.multi INFO: found job 16914762 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914762 on provider checkpoint proc_lsst.multi INFO: cancelling 16914762 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914763 in provider checkpoint proc_lsst.multi INFO: found job 16914763 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914763 on provider checkpoint proc_lsst.multi INFO: cancelling 16914763 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914764 in provider checkpoint proc_lsst.multi INFO: found job 16914764 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914764 on provider checkpoint proc_lsst.multi INFO: cancelling 16914764 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914765 in provider checkpoint proc_lsst.multi INFO: found job 16914765 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914765 on provider checkpoint proc_lsst.multi INFO: cancelling 16914765 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914766 in provider checkpoint proc_lsst.multi INFO: found job 16914766 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914766 on provider checkpoint proc_lsst.multi INFO: cancelling 16914766 on provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:146 [INFO] found job 16914767 in provider checkpoint proc_lsst.multi INFO: found job 16914767 in provider checkpoint 2024-03-08 10:03:23 proc_lsst.multi:201 [INFO] cancelling 16914767 on provider checkpoint proc_lsst.multi INFO: cancelling 16914767 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914768 in provider checkpoint proc_lsst.multi INFO: found job 16914768 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914768 on provider checkpoint proc_lsst.multi INFO: cancelling 16914768 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914769 in provider checkpoint proc_lsst.multi INFO: found job 16914769 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914769 on provider checkpoint proc_lsst.multi INFO: cancelling 16914769 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914770 in provider checkpoint proc_lsst.multi INFO: found job 16914770 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914770 on provider checkpoint proc_lsst.multi INFO: cancelling 16914770 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914771 in provider checkpoint proc_lsst.multi INFO: found job 16914771 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914771 on provider checkpoint proc_lsst.multi INFO: cancelling 16914771 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914772 in provider checkpoint proc_lsst.multi INFO: found job 16914772 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914772 on provider checkpoint proc_lsst.multi INFO: cancelling 16914772 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914773 in provider checkpoint proc_lsst.multi INFO: found job 16914773 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914773 on provider checkpoint proc_lsst.multi INFO: cancelling 16914773 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914774 in provider checkpoint proc_lsst.multi INFO: found job 16914774 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914774 on provider checkpoint proc_lsst.multi INFO: cancelling 16914774 on provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:146 [INFO] found job 16914775 in provider checkpoint proc_lsst.multi INFO: found job 16914775 in provider checkpoint 2024-03-08 10:03:24 proc_lsst.multi:201 [INFO] cancelling 16914775 on provider checkpoint proc_lsst.multi INFO: cancelling 16914775 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914776 in provider checkpoint proc_lsst.multi INFO: found job 16914776 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914776 on provider checkpoint proc_lsst.multi INFO: cancelling 16914776 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914777 in provider checkpoint proc_lsst.multi INFO: found job 16914777 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914777 on provider checkpoint proc_lsst.multi INFO: cancelling 16914777 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914778 in provider checkpoint proc_lsst.multi INFO: found job 16914778 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914778 on provider checkpoint proc_lsst.multi INFO: cancelling 16914778 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914779 in provider checkpoint proc_lsst.multi INFO: found job 16914779 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914779 on provider checkpoint proc_lsst.multi INFO: cancelling 16914779 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914780 in provider checkpoint proc_lsst.multi INFO: found job 16914780 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914780 on provider checkpoint proc_lsst.multi INFO: cancelling 16914780 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914781 in provider checkpoint proc_lsst.multi INFO: found job 16914781 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914781 on provider checkpoint proc_lsst.multi INFO: cancelling 16914781 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914782 in provider checkpoint proc_lsst.multi INFO: found job 16914782 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914782 on provider checkpoint proc_lsst.multi INFO: cancelling 16914782 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914783 in provider checkpoint proc_lsst.multi INFO: found job 16914783 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914783 on provider checkpoint proc_lsst.multi INFO: cancelling 16914783 on provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:146 [INFO] found job 16914784 in provider checkpoint proc_lsst.multi INFO: found job 16914784 in provider checkpoint 2024-03-08 10:03:25 proc_lsst.multi:201 [INFO] cancelling 16914784 on provider checkpoint proc_lsst.multi INFO: cancelling 16914784 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914785 in provider checkpoint proc_lsst.multi INFO: found job 16914785 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914785 on provider checkpoint proc_lsst.multi INFO: cancelling 16914785 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914786 in provider checkpoint proc_lsst.multi INFO: found job 16914786 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914786 on provider checkpoint proc_lsst.multi INFO: cancelling 16914786 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914787 in provider checkpoint proc_lsst.multi INFO: found job 16914787 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914787 on provider checkpoint proc_lsst.multi INFO: cancelling 16914787 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914788 in provider checkpoint proc_lsst.multi INFO: found job 16914788 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914788 on provider checkpoint proc_lsst.multi INFO: cancelling 16914788 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914789 in provider checkpoint proc_lsst.multi INFO: found job 16914789 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914789 on provider checkpoint proc_lsst.multi INFO: cancelling 16914789 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914790 in provider checkpoint proc_lsst.multi INFO: found job 16914790 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914790 on provider checkpoint proc_lsst.multi INFO: cancelling 16914790 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914791 in provider checkpoint proc_lsst.multi INFO: found job 16914791 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914791 on provider checkpoint proc_lsst.multi INFO: cancelling 16914791 on provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:146 [INFO] found job 16914792 in provider checkpoint proc_lsst.multi INFO: found job 16914792 in provider checkpoint 2024-03-08 10:03:26 proc_lsst.multi:201 [INFO] cancelling 16914792 on provider checkpoint proc_lsst.multi INFO: cancelling 16914792 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914793 in provider checkpoint proc_lsst.multi INFO: found job 16914793 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914793 on provider checkpoint proc_lsst.multi INFO: cancelling 16914793 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914794 in provider checkpoint proc_lsst.multi INFO: found job 16914794 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914794 on provider checkpoint proc_lsst.multi INFO: cancelling 16914794 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914800 in provider checkpoint proc_lsst.multi INFO: found job 16914800 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914800 on provider checkpoint proc_lsst.multi INFO: cancelling 16914800 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914801 in provider checkpoint proc_lsst.multi INFO: found job 16914801 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914801 on provider checkpoint proc_lsst.multi INFO: cancelling 16914801 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914802 in provider checkpoint proc_lsst.multi INFO: found job 16914802 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914802 on provider checkpoint proc_lsst.multi INFO: cancelling 16914802 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914803 in provider checkpoint proc_lsst.multi INFO: found job 16914803 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914803 on provider checkpoint proc_lsst.multi INFO: cancelling 16914803 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914804 in provider checkpoint proc_lsst.multi INFO: found job 16914804 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914804 on provider checkpoint proc_lsst.multi INFO: cancelling 16914804 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914805 in provider checkpoint proc_lsst.multi INFO: found job 16914805 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914805 on provider checkpoint proc_lsst.multi INFO: cancelling 16914805 on provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:146 [INFO] found job 16914806 in provider checkpoint proc_lsst.multi INFO: found job 16914806 in provider checkpoint 2024-03-08 10:03:27 proc_lsst.multi:201 [INFO] cancelling 16914806 on provider checkpoint proc_lsst.multi INFO: cancelling 16914806 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914807 in provider checkpoint proc_lsst.multi INFO: found job 16914807 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914807 on provider checkpoint proc_lsst.multi INFO: cancelling 16914807 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914808 in provider checkpoint proc_lsst.multi INFO: found job 16914808 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914808 on provider checkpoint proc_lsst.multi INFO: cancelling 16914808 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914809 in provider checkpoint proc_lsst.multi INFO: found job 16914809 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914809 on provider checkpoint proc_lsst.multi INFO: cancelling 16914809 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914810 in provider checkpoint proc_lsst.multi INFO: found job 16914810 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914810 on provider checkpoint proc_lsst.multi INFO: cancelling 16914810 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914811 in provider checkpoint proc_lsst.multi INFO: found job 16914811 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914811 on provider checkpoint proc_lsst.multi INFO: cancelling 16914811 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914812 in provider checkpoint proc_lsst.multi INFO: found job 16914812 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914812 on provider checkpoint proc_lsst.multi INFO: cancelling 16914812 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914813 in provider checkpoint proc_lsst.multi INFO: found job 16914813 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914813 on provider checkpoint proc_lsst.multi INFO: cancelling 16914813 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914814 in provider checkpoint proc_lsst.multi INFO: found job 16914814 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914814 on provider checkpoint proc_lsst.multi INFO: cancelling 16914814 on provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:146 [INFO] found job 16914815 in provider checkpoint proc_lsst.multi INFO: found job 16914815 in provider checkpoint 2024-03-08 10:03:28 proc_lsst.multi:201 [INFO] cancelling 16914815 on provider checkpoint proc_lsst.multi INFO: cancelling 16914815 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914816 in provider checkpoint proc_lsst.multi INFO: found job 16914816 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914816 on provider checkpoint proc_lsst.multi INFO: cancelling 16914816 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914817 in provider checkpoint proc_lsst.multi INFO: found job 16914817 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914817 on provider checkpoint proc_lsst.multi INFO: cancelling 16914817 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914818 in provider checkpoint proc_lsst.multi INFO: found job 16914818 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914818 on provider checkpoint proc_lsst.multi INFO: cancelling 16914818 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914819 in provider checkpoint proc_lsst.multi INFO: found job 16914819 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914819 on provider checkpoint proc_lsst.multi INFO: cancelling 16914819 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914820 in provider checkpoint proc_lsst.multi INFO: found job 16914820 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914820 on provider checkpoint proc_lsst.multi INFO: cancelling 16914820 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914821 in provider checkpoint proc_lsst.multi INFO: found job 16914821 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914821 on provider checkpoint proc_lsst.multi INFO: cancelling 16914821 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914822 in provider checkpoint proc_lsst.multi INFO: found job 16914822 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914822 on provider checkpoint proc_lsst.multi INFO: cancelling 16914822 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914823 in provider checkpoint proc_lsst.multi INFO: found job 16914823 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914823 on provider checkpoint proc_lsst.multi INFO: cancelling 16914823 on provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:146 [INFO] found job 16914824 in provider checkpoint proc_lsst.multi INFO: found job 16914824 in provider checkpoint 2024-03-08 10:03:29 proc_lsst.multi:201 [INFO] cancelling 16914824 on provider checkpoint proc_lsst.multi INFO: cancelling 16914824 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914825 in provider checkpoint proc_lsst.multi INFO: found job 16914825 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914825 on provider checkpoint proc_lsst.multi INFO: cancelling 16914825 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914826 in provider checkpoint proc_lsst.multi INFO: found job 16914826 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914826 on provider checkpoint proc_lsst.multi INFO: cancelling 16914826 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914827 in provider checkpoint proc_lsst.multi INFO: found job 16914827 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914827 on provider checkpoint proc_lsst.multi INFO: cancelling 16914827 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914828 in provider checkpoint proc_lsst.multi INFO: found job 16914828 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914828 on provider checkpoint proc_lsst.multi INFO: cancelling 16914828 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914829 in provider checkpoint proc_lsst.multi INFO: found job 16914829 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914829 on provider checkpoint proc_lsst.multi INFO: cancelling 16914829 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914830 in provider checkpoint proc_lsst.multi INFO: found job 16914830 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914830 on provider checkpoint proc_lsst.multi INFO: cancelling 16914830 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914831 in provider checkpoint proc_lsst.multi INFO: found job 16914831 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914831 on provider checkpoint proc_lsst.multi INFO: cancelling 16914831 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914832 in provider checkpoint proc_lsst.multi INFO: found job 16914832 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914832 on provider checkpoint proc_lsst.multi INFO: cancelling 16914832 on provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:146 [INFO] found job 16914833 in provider checkpoint proc_lsst.multi INFO: found job 16914833 in provider checkpoint 2024-03-08 10:03:30 proc_lsst.multi:201 [INFO] cancelling 16914833 on provider checkpoint proc_lsst.multi INFO: cancelling 16914833 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914834 in provider checkpoint proc_lsst.multi INFO: found job 16914834 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914834 on provider checkpoint proc_lsst.multi INFO: cancelling 16914834 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914835 in provider checkpoint proc_lsst.multi INFO: found job 16914835 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914835 on provider checkpoint proc_lsst.multi INFO: cancelling 16914835 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914836 in provider checkpoint proc_lsst.multi INFO: found job 16914836 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914836 on provider checkpoint proc_lsst.multi INFO: cancelling 16914836 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914837 in provider checkpoint proc_lsst.multi INFO: found job 16914837 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914837 on provider checkpoint proc_lsst.multi INFO: cancelling 16914837 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914838 in provider checkpoint proc_lsst.multi INFO: found job 16914838 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914838 on provider checkpoint proc_lsst.multi INFO: cancelling 16914838 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914839 in provider checkpoint proc_lsst.multi INFO: found job 16914839 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914839 on provider checkpoint proc_lsst.multi INFO: cancelling 16914839 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914840 in provider checkpoint proc_lsst.multi INFO: found job 16914840 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914840 on provider checkpoint proc_lsst.multi INFO: cancelling 16914840 on provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:146 [INFO] found job 16914841 in provider checkpoint proc_lsst.multi INFO: found job 16914841 in provider checkpoint 2024-03-08 10:03:31 proc_lsst.multi:201 [INFO] cancelling 16914841 on provider checkpoint proc_lsst.multi INFO: cancelling 16914841 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914842 in provider checkpoint proc_lsst.multi INFO: found job 16914842 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914842 on provider checkpoint proc_lsst.multi INFO: cancelling 16914842 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914843 in provider checkpoint proc_lsst.multi INFO: found job 16914843 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914843 on provider checkpoint proc_lsst.multi INFO: cancelling 16914843 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914844 in provider checkpoint proc_lsst.multi INFO: found job 16914844 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914844 on provider checkpoint proc_lsst.multi INFO: cancelling 16914844 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914845 in provider checkpoint proc_lsst.multi INFO: found job 16914845 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914845 on provider checkpoint proc_lsst.multi INFO: cancelling 16914845 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914846 in provider checkpoint proc_lsst.multi INFO: found job 16914846 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914846 on provider checkpoint proc_lsst.multi INFO: cancelling 16914846 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914847 in provider checkpoint proc_lsst.multi INFO: found job 16914847 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914847 on provider checkpoint proc_lsst.multi INFO: cancelling 16914847 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914848 in provider checkpoint proc_lsst.multi INFO: found job 16914848 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914848 on provider checkpoint proc_lsst.multi INFO: cancelling 16914848 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914849 in provider checkpoint proc_lsst.multi INFO: found job 16914849 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914849 on provider checkpoint proc_lsst.multi INFO: cancelling 16914849 on provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:146 [INFO] found job 16914850 in provider checkpoint proc_lsst.multi INFO: found job 16914850 in provider checkpoint 2024-03-08 10:03:32 proc_lsst.multi:201 [INFO] cancelling 16914850 on provider checkpoint proc_lsst.multi INFO: cancelling 16914850 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914851 in provider checkpoint proc_lsst.multi INFO: found job 16914851 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914851 on provider checkpoint proc_lsst.multi INFO: cancelling 16914851 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914852 in provider checkpoint proc_lsst.multi INFO: found job 16914852 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914852 on provider checkpoint proc_lsst.multi INFO: cancelling 16914852 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914853 in provider checkpoint proc_lsst.multi INFO: found job 16914853 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914853 on provider checkpoint proc_lsst.multi INFO: cancelling 16914853 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914854 in provider checkpoint proc_lsst.multi INFO: found job 16914854 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914854 on provider checkpoint proc_lsst.multi INFO: cancelling 16914854 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914855 in provider checkpoint proc_lsst.multi INFO: found job 16914855 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914855 on provider checkpoint proc_lsst.multi INFO: cancelling 16914855 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914856 in provider checkpoint proc_lsst.multi INFO: found job 16914856 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914856 on provider checkpoint proc_lsst.multi INFO: cancelling 16914856 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914857 in provider checkpoint proc_lsst.multi INFO: found job 16914857 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914857 on provider checkpoint proc_lsst.multi INFO: cancelling 16914857 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914858 in provider checkpoint proc_lsst.multi INFO: found job 16914858 in provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914858 on provider checkpoint proc_lsst.multi INFO: cancelling 16914858 on provider checkpoint 2024-03-08 10:03:33 proc_lsst.multi:146 [INFO] found job 16914600 in provider astro proc_lsst.multi INFO: found job 16914600 in provider astro 2024-03-08 10:03:33 proc_lsst.multi:201 [INFO] cancelling 16914600 on provider astro proc_lsst.multi INFO: cancelling 16914600 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914601 in provider astro proc_lsst.multi INFO: found job 16914601 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914601 on provider astro proc_lsst.multi INFO: cancelling 16914601 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914602 in provider astro proc_lsst.multi INFO: found job 16914602 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914602 on provider astro proc_lsst.multi INFO: cancelling 16914602 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914603 in provider astro proc_lsst.multi INFO: found job 16914603 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914603 on provider astro proc_lsst.multi INFO: cancelling 16914603 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914604 in provider astro proc_lsst.multi INFO: found job 16914604 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914604 on provider astro proc_lsst.multi INFO: cancelling 16914604 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914605 in provider astro proc_lsst.multi INFO: found job 16914605 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914605 on provider astro proc_lsst.multi INFO: cancelling 16914605 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914606 in provider astro proc_lsst.multi INFO: found job 16914606 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914606 on provider astro proc_lsst.multi INFO: cancelling 16914606 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914607 in provider astro proc_lsst.multi INFO: found job 16914607 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914607 on provider astro proc_lsst.multi INFO: cancelling 16914607 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914608 in provider astro proc_lsst.multi INFO: found job 16914608 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914608 on provider astro proc_lsst.multi INFO: cancelling 16914608 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914609 in provider astro proc_lsst.multi INFO: found job 16914609 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914609 on provider astro proc_lsst.multi INFO: cancelling 16914609 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914610 in provider astro proc_lsst.multi INFO: found job 16914610 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914610 on provider astro proc_lsst.multi INFO: cancelling 16914610 on provider astro 2024-03-08 10:03:34 proc_lsst.multi:146 [INFO] found job 16914611 in provider astro proc_lsst.multi INFO: found job 16914611 in provider astro 2024-03-08 10:03:34 proc_lsst.multi:201 [INFO] cancelling 16914611 on provider astro proc_lsst.multi INFO: cancelling 16914611 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914612 in provider astro proc_lsst.multi INFO: found job 16914612 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914612 on provider astro proc_lsst.multi INFO: cancelling 16914612 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914613 in provider astro proc_lsst.multi INFO: found job 16914613 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914613 on provider astro proc_lsst.multi INFO: cancelling 16914613 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914614 in provider astro proc_lsst.multi INFO: found job 16914614 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914614 on provider astro proc_lsst.multi INFO: cancelling 16914614 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914615 in provider astro proc_lsst.multi INFO: found job 16914615 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914615 on provider astro proc_lsst.multi INFO: cancelling 16914615 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914616 in provider astro proc_lsst.multi INFO: found job 16914616 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914616 on provider astro proc_lsst.multi INFO: cancelling 16914616 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914617 in provider astro proc_lsst.multi INFO: found job 16914617 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914617 on provider astro proc_lsst.multi INFO: cancelling 16914617 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914618 in provider astro proc_lsst.multi INFO: found job 16914618 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914618 on provider astro proc_lsst.multi INFO: cancelling 16914618 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914619 in provider astro proc_lsst.multi INFO: found job 16914619 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914619 on provider astro proc_lsst.multi INFO: cancelling 16914619 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914620 in provider astro proc_lsst.multi INFO: found job 16914620 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914620 on provider astro proc_lsst.multi INFO: cancelling 16914620 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914621 in provider astro proc_lsst.multi INFO: found job 16914621 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914621 on provider astro proc_lsst.multi INFO: cancelling 16914621 on provider astro 2024-03-08 10:03:35 proc_lsst.multi:146 [INFO] found job 16914622 in provider astro proc_lsst.multi INFO: found job 16914622 in provider astro 2024-03-08 10:03:35 proc_lsst.multi:201 [INFO] cancelling 16914622 on provider astro proc_lsst.multi INFO: cancelling 16914622 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914623 in provider astro proc_lsst.multi INFO: found job 16914623 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914623 on provider astro proc_lsst.multi INFO: cancelling 16914623 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914624 in provider astro proc_lsst.multi INFO: found job 16914624 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914624 on provider astro proc_lsst.multi INFO: cancelling 16914624 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914625 in provider astro proc_lsst.multi INFO: found job 16914625 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914625 on provider astro proc_lsst.multi INFO: cancelling 16914625 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914626 in provider astro proc_lsst.multi INFO: found job 16914626 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914626 on provider astro proc_lsst.multi INFO: cancelling 16914626 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914627 in provider astro proc_lsst.multi INFO: found job 16914627 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914627 on provider astro proc_lsst.multi INFO: cancelling 16914627 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914628 in provider astro proc_lsst.multi INFO: found job 16914628 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914628 on provider astro proc_lsst.multi INFO: cancelling 16914628 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914629 in provider astro proc_lsst.multi INFO: found job 16914629 in provider astro 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914629 on provider astro proc_lsst.multi INFO: cancelling 16914629 on provider astro 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914635 in provider escience_a40 proc_lsst.multi INFO: found job 16914635 in provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914635 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914635 on provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914636 in provider escience_a40 proc_lsst.multi INFO: found job 16914636 in provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914636 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914636 on provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914637 in provider escience_a40 proc_lsst.multi INFO: found job 16914637 in provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914637 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914637 on provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:146 [INFO] found job 16914638 in provider escience_a40 proc_lsst.multi INFO: found job 16914638 in provider escience_a40 2024-03-08 10:03:36 proc_lsst.multi:201 [INFO] cancelling 16914638 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914638 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914639 in provider escience_a40 proc_lsst.multi INFO: found job 16914639 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914639 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914639 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914640 in provider escience_a40 proc_lsst.multi INFO: found job 16914640 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914640 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914640 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914641 in provider escience_a40 proc_lsst.multi INFO: found job 16914641 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914641 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914641 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914642 in provider escience_a40 proc_lsst.multi INFO: found job 16914642 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914642 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914642 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914643 in provider escience_a40 proc_lsst.multi INFO: found job 16914643 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914643 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914643 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914647 in provider escience_a40 proc_lsst.multi INFO: found job 16914647 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914647 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914647 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914648 in provider escience_a40 proc_lsst.multi INFO: found job 16914648 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914648 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914648 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914649 in provider escience_a40 proc_lsst.multi INFO: found job 16914649 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914649 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914649 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914650 in provider escience_a40 proc_lsst.multi INFO: found job 16914650 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914650 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914650 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914651 in provider escience_a40 proc_lsst.multi INFO: found job 16914651 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914651 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914651 on provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:146 [INFO] found job 16914652 in provider escience_a40 proc_lsst.multi INFO: found job 16914652 in provider escience_a40 2024-03-08 10:03:37 proc_lsst.multi:201 [INFO] cancelling 16914652 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914652 on provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914654 in provider escience_a40 proc_lsst.multi INFO: found job 16914654 in provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914654 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914654 on provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914655 in provider escience_a40 proc_lsst.multi INFO: found job 16914655 in provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914655 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914655 on provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914656 in provider escience_a40 proc_lsst.multi INFO: found job 16914656 in provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914656 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914656 on provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914657 in provider escience_a40 proc_lsst.multi INFO: found job 16914657 in provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914657 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914657 on provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914658 in provider escience_a40 proc_lsst.multi INFO: found job 16914658 in provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914658 on provider escience_a40 proc_lsst.multi INFO: cancelling 16914658 on provider escience_a40 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914630 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914630 in provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914630 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914630 on provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914631 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914631 in provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914631 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914631 on provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914632 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914632 in provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914632 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914632 on provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914633 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914633 in provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914633 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914633 on provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 16914634 in provider escience_rtx6k proc_lsst.multi INFO: found job 16914634 in provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 16914634 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16914634 on provider escience_rtx6k 2024-03-08 10:03:38 proc_lsst.multi:146 [INFO] found job 30716 in provider local proc_lsst.multi INFO: found job 30716 in provider local 2024-03-08 10:03:38 proc_lsst.multi:201 [INFO] cancelling 30716 on provider local proc_lsst.multi INFO: cancelling 30716 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 30716 parsl.providers.local.local WARNING: Failed to kill PID: 30716 and child processes on local 2024-03-08 10:03:39 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 193.8379 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190504_A1c_science#step0_20240308T100021Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 193.8448 seconds; current memory usage: 0.289 Gibyte, delta: 0.007 Gibyte, peak delta: 0.008 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 205.6074 seconds; current memory usage: 0.289 Gibyte, delta: 0.152 Gibyte, peak delta: 0.153 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.289 Gibyte (main), 0.289 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190504_A1c_science#step0_20240308T100021Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 3m27.562s user 0m58.265s sys 0m30.790s