+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20190928/B1a --output-run DEEP/20190928/B1a/science#step0/20240313T131829Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.7332 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0040 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0069 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/pipeline_orig.qgraph DEEP/20190928/B1a/science#step0/20240313T131829Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.8831 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.9665 seconds; current memory usage: 0.272 Gibyte, delta: 0.094 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.266 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0135 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.266 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20190928_B1a_science#step0_20240313T131829Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0807 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.266 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2355 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.266 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 1ea2c919-c23b-438e-af21-fb54abbbf39b parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-13 13:18:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-13 13:18:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 13:18:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-13 13:18:55 proc_lsst.multi:170 [INFO] [multi] job_id 86094 proc_lsst.multi INFO: [multi] job_id 86094 2024-03-13 13:18:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:55 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 86094 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 86094 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 86094 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 10ba46bb08967c6afb38d7edc88f7a9f parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d613ce4b-b229-4f96-a048-8063f5439b16_isrForCrosstalkSources_899014_33.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d613ce4b-b229-4f96-a048-8063f5439b16_isrForCrosstalkSources_899014_33.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash a5510354f30b87edba4430c5d902e8ee parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/542f68b2-a3d6-40c6-93c7-7fda584f61ba_isrForCrosstalkSources_899118_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/542f68b2-a3d6-40c6-93c7-7fda584f61ba_isrForCrosstalkSources_899118_27.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 424283d585070cd2fefbef4b5f976815 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f637bb7d-9768-498e-a8b5-508367832351_isrForCrosstalkSources_899016_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f637bb7d-9768-498e-a8b5-508367832351_isrForCrosstalkSources_899016_28.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 0b5f4c907285d28af6d26e3a85aae0dd parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0c3bc967-b10c-461a-b5a7-67623945cf32_isrForCrosstalkSources_899118_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0c3bc967-b10c-461a-b5a7-67623945cf32_isrForCrosstalkSources_899118_62.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 3601fd727c651b0d4ab2c95e8716de37 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/877eb286-d6f5-45f4-8f77-04d9f8bfd044_isrForCrosstalkSources_899015_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/877eb286-d6f5-45f4-8f77-04d9f8bfd044_isrForCrosstalkSources_899015_62.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash d97206251843ccd69cda06229036db84 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb418a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ed8608df-5026-4e7d-9d66-7d16a63c53e7_isrForCrosstalkSources_899117_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ed8608df-5026-4e7d-9d66-7d16a63c53e7_isrForCrosstalkSources_899117_43.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 9c88f920c0656ef308f7edec1c148979 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/62c435a4-7287-4f60-b848-637dff1d04f2_isrForCrosstalkSources_899014_7.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/62c435a4-7287-4f60-b848-637dff1d04f2_isrForCrosstalkSources_899014_7.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash f95a8e350ac7be23ae3d6200bcbd6b7d parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb40c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9e511fd3-70a0-466a-a0c8-e467fbb7cc95_isrForCrosstalkSources_899119_31.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9e511fd3-70a0-466a-a0c8-e467fbb7cc95_isrForCrosstalkSources_899119_31.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash c03dba1ad4ff1f75041a3337b5beb8a2 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a2963aed-25a2-48dc-8a32-a0afc7131109_isrForCrosstalkSources_899117_1.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a2963aed-25a2-48dc-8a32-a0afc7131109_isrForCrosstalkSources_899117_1.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 74d3c8d0ce9807ad5dfbe01997db9e71 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb414e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/10a38e77-085b-4a26-a1b2-49947617fee1_isrForCrosstalkSources_899014_43.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/10a38e77-085b-4a26-a1b2-49947617fee1_isrForCrosstalkSources_899014_43.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 4320a304a771da3d1a0310dbce0fd813 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb419e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d978f10f-59e5-4b47-a575-66225db035ad_isrForCrosstalkSources_899119_40.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d978f10f-59e5-4b47-a575-66225db035ad_isrForCrosstalkSources_899119_40.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash b58c5f20210cbf9b45973d7708a116ed parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/09d20420-df7b-4c2c-a302-0b1c37dd4ebb_isrForCrosstalkSources_899014_30.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/09d20420-df7b-4c2c-a302-0b1c37dd4ebb_isrForCrosstalkSources_899014_30.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 0e4d72c406f33eb49a72e1cbafaf4fa0 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/b611d768-f394-4abb-b91b-11d2f95ee218_isrForCrosstalkSources_899117_56.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/b611d768-f394-4abb-b91b-11d2f95ee218_isrForCrosstalkSources_899117_56.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash b489920628722c9c1a143e0941920732 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f6798385-2e83-4853-8e7a-7245ae87954a_isrForCrosstalkSources_899015_59.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f6798385-2e83-4853-8e7a-7245ae87954a_isrForCrosstalkSources_899015_59.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 7cf8db8af66fa220f0909a015bf85592 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/efb31862-e44d-468c-83a6-a303ee6c738d_isrForCrosstalkSources_899117_36.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/efb31862-e44d-468c-83a6-a303ee6c738d_isrForCrosstalkSources_899117_36.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 7166137a3dff62fda33ec95d0e0a05ba parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb41ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/93ec4364-0aa5-4547-8f0c-2fbfde7f4dcf_isrForCrosstalkSources_899014_25.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/93ec4364-0aa5-4547-8f0c-2fbfde7f4dcf_isrForCrosstalkSources_899014_25.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 55133e5ac2f056027cb4406944dd2567 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb423e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f5a9544e-0955-432b-9fc6-6bfefef45e9d_isrForCrosstalkSources_899118_16.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f5a9544e-0955-432b-9fc6-6bfefef45e9d_isrForCrosstalkSources_899118_16.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 6a576d73bf884cd6f4aa7b11201918c8 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb425c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ecef1c84-c53a-4f99-9a7f-7672047e7b38_isrForCrosstalkSources_899118_13.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ecef1c84-c53a-4f99-9a7f-7672047e7b38_isrForCrosstalkSources_899118_13.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 6a3992dc91c6c0662661a1cf53f347e5 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d9af716-134c-4180-b22b-f6df54ffcc2e_isrForCrosstalkSources_899117_44.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d9af716-134c-4180-b22b-f6df54ffcc2e_isrForCrosstalkSources_899117_44.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 1415b747160d1f85c26b6010c69a7d22 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f5394aeb-c4ef-4123-9338-5fa12bf696ef_isrForCrosstalkSources_899117_60.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f5394aeb-c4ef-4123-9338-5fa12bf696ef_isrForCrosstalkSources_899117_60.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash a351fa01bfc5c599397c1f7209a75548 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb422a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d2c144f6-6547-453e-8915-fde85522763c_isrForCrosstalkSources_899117_35.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d2c144f6-6547-453e-8915-fde85522763c_isrForCrosstalkSources_899117_35.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 9efa05f0b167c41036462960bec9cc1e parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28f03f4c-2780-4275-b22c-5eef68520cb4_isrForCrosstalkSources_899014_35.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28f03f4c-2780-4275-b22c-5eef68520cb4_isrForCrosstalkSources_899014_35.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash def1015d58dda7a4f0e4da9eb36d9423 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e104b3b0-2d69-4716-98e4-b3b3b079877f_isrForCrosstalkSources_899014_40.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e104b3b0-2d69-4716-98e4-b3b3b079877f_isrForCrosstalkSources_899014_40.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash cca636bc57f1200ac2a32359252942f8 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb427a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/10097c6f-1206-499f-bf45-1737b480136d_isrForCrosstalkSources_899117_3.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/10097c6f-1206-499f-bf45-1737b480136d_isrForCrosstalkSources_899117_3.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 674d9f083232f34252f3590e3a562199 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/79fd6eef-afa3-4afa-9417-b2ab12bea07c_isrForCrosstalkSources_899015_53.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/79fd6eef-afa3-4afa-9417-b2ab12bea07c_isrForCrosstalkSources_899015_53.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 2b0582266ec6c2af7e1ba1682db1f577 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d22dd43f-7bb8-44c7-8a97-cc2a873e3208_isrForCrosstalkSources_899117_4.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d22dd43f-7bb8-44c7-8a97-cc2a873e3208_isrForCrosstalkSources_899117_4.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 087245b411325b0a6c7aa7c1f6393522 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb428e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8fd587ab-3940-41db-a2ee-4c22417e597e_isrForCrosstalkSources_899014_27.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8fd587ab-3940-41db-a2ee-4c22417e597e_isrForCrosstalkSources_899014_27.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 863dffeb07a8e0e4b1eb6154205af3f5 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c91dff3a-c5a9-4053-b12c-51ea843c5db4_isrForCrosstalkSources_899015_33.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c91dff3a-c5a9-4053-b12c-51ea843c5db4_isrForCrosstalkSources_899015_33.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 9d484c044020e83071069564633dd72a parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0cedd62e-0d9f-4308-ae72-a42fc39a683c_isrForCrosstalkSources_899119_33.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0cedd62e-0d9f-4308-ae72-a42fc39a683c_isrForCrosstalkSources_899119_33.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash c5f715a9a7295c38cf8cff1f96d0733e parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4a3b0460-7d21-460f-8679-8e55e3d857fa_isrForCrosstalkSources_899117_27.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4a3b0460-7d21-460f-8679-8e55e3d857fa_isrForCrosstalkSources_899117_27.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 92a957331330c531b0f92733fc23e1e8 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/1a597015-7484-405f-b243-47eda7bbbd89_isrForCrosstalkSources_899014_45.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/1a597015-7484-405f-b243-47eda7bbbd89_isrForCrosstalkSources_899014_45.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 3c8ff39c570ba8eeea50d96107db84da parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/69c8361d-e85a-4303-bded-10122edfc057_isrForCrosstalkSources_899015_13.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/69c8361d-e85a-4303-bded-10122edfc057_isrForCrosstalkSources_899015_13.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 02e29e01f85ec601f3d99ae1abb84371 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0894dfae-4342-4175-8352-df113042dce3_isrForCrosstalkSources_899118_55.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0894dfae-4342-4175-8352-df113042dce3_isrForCrosstalkSources_899118_55.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash a731aea95885fa770a94a22fd82ffe94 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b6d302fb-fb36-48b2-bfed-16e07370d2ce_isrForCrosstalkSources_899119_22.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b6d302fb-fb36-48b2-bfed-16e07370d2ce_isrForCrosstalkSources_899119_22.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 9c892935a88fd5fc8ac3b13d079f7605 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f59fc51b-010d-4605-a579-70f45c895829_isrForCrosstalkSources_899117_48.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f59fc51b-010d-4605-a579-70f45c895829_isrForCrosstalkSources_899117_48.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash a8e992b389f46d7d686f7774cb04b1af parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8d87e1a1-18e5-4e2d-8047-d5bb831ab705_isrForCrosstalkSources_899015_4.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8d87e1a1-18e5-4e2d-8047-d5bb831ab705_isrForCrosstalkSources_899015_4.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash f406e606f3ccf4e357b0072770357bb5 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ce705733-2c39-4262-bb9a-f305df88b554_isrForCrosstalkSources_899118_24.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ce705733-2c39-4262-bb9a-f305df88b554_isrForCrosstalkSources_899118_24.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 5406f371ecbbeab81e0e172af77a9b69 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb431a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c0cb8ecc-6e06-4af6-98d3-511a2a4388bb_isrForCrosstalkSources_899119_14.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c0cb8ecc-6e06-4af6-98d3-511a2a4388bb_isrForCrosstalkSources_899119_14.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 63038df18c5d0351e47a43206d4ade82 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d05221e4-8f7d-4379-a552-ddba1037e4a8_isrForCrosstalkSources_899118_29.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d05221e4-8f7d-4379-a552-ddba1037e4a8_isrForCrosstalkSources_899118_29.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 48580a4f84922627c2550d99d625aea8 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1d55e837-d9c8-4a56-ba24-dee49151c5ed_isrForCrosstalkSources_899016_34.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1d55e837-d9c8-4a56-ba24-dee49151c5ed_isrForCrosstalkSources_899016_34.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 53a61472e65aef45c267369e3d09cce8 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/102a342d-7e49-4948-b094-dccb3d122e71_isrForCrosstalkSources_899014_51.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/102a342d-7e49-4948-b094-dccb3d122e71_isrForCrosstalkSources_899014_51.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash dcd67ffbb7036e75a8c7b443c0c5ce21 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ffbc126b-1527-40b6-9d86-8d32e25d3ac7_isrForCrosstalkSources_899117_33.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ffbc126b-1527-40b6-9d86-8d32e25d3ac7_isrForCrosstalkSources_899117_33.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash c724f790f639c3420f36559716621a41 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/a2f138aa-4514-4369-b772-c77a4199e545_isrForCrosstalkSources_899014_42.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/a2f138aa-4514-4369-b772-c77a4199e545_isrForCrosstalkSources_899014_42.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash c01c84d48b5eb91943c052395d536289 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/dc8eec57-1d7b-42e9-8adb-bdfdfa478137_isrForCrosstalkSources_899016_39.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/dc8eec57-1d7b-42e9-8adb-bdfdfa478137_isrForCrosstalkSources_899016_39.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 1a9ddbc28ea7af42a6d9ff8d5510c9a4 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/eae04f38-b94b-441a-815c-d27c6cf0d7ba_isrForCrosstalkSources_899015_36.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/eae04f38-b94b-441a-815c-d27c6cf0d7ba_isrForCrosstalkSources_899015_36.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 73e24406fd71cea6e0df021d112f58c7 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7c9b01b3-bac2-425d-820f-eebaef40f5c2_isrForCrosstalkSources_899117_28.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7c9b01b3-bac2-425d-820f-eebaef40f5c2_isrForCrosstalkSources_899117_28.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 43a08380326e1813962b3bc4123cc771 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca206f57-afa7-406c-b730-d248ef51b53e_isrForCrosstalkSources_899016_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca206f57-afa7-406c-b730-d248ef51b53e_isrForCrosstalkSources_899016_61.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 4455b6f0b8db9fbc8ff34d7c4a7fe101 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb439c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a4f6f04-98be-4f40-a529-fe6c05490ec3_isrForCrosstalkSources_899119_12.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a4f6f04-98be-4f40-a529-fe6c05490ec3_isrForCrosstalkSources_899119_12.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash cc1c5811f02e2aab327feef17daf36fb parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/fd08e855-e5d0-4d75-a4f7-9bd9de155cf7_isrForCrosstalkSources_899119_39.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/fd08e855-e5d0-4d75-a4f7-9bd9de155cf7_isrForCrosstalkSources_899119_39.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 49ec9b7ec32685592b4fa53c21b71905 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb437e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/56774c2f-decd-4dc8-b359-f987bba68a88_isrForCrosstalkSources_899117_14.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/56774c2f-decd-4dc8-b359-f987bba68a88_isrForCrosstalkSources_899117_14.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 9ac66f4cd85fd845569339ece4187e5a parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5ee9088f-308b-4574-9f68-8846045f7532_isrForCrosstalkSources_899118_3.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5ee9088f-308b-4574-9f68-8846045f7532_isrForCrosstalkSources_899118_3.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 6712ace11d8643f8964b6a54e1c3be67 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c98c1e92-2110-43c8-92b4-04e0bf16827b_isrForCrosstalkSources_899119_56.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c98c1e92-2110-43c8-92b4-04e0bf16827b_isrForCrosstalkSources_899119_56.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 4b146cbdeb593e9a6bd043eb91517e35 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb436a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/cffc49ef-2481-4746-a4df-dbf4c453e689_isrForCrosstalkSources_899117_45.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/cffc49ef-2481-4746-a4df-dbf4c453e689_isrForCrosstalkSources_899117_45.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash a9854d4755b4a1a285490a6248c988ed parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/1058b3f3-333a-476a-86b8-975c3ceb9d9f_isrForCrosstalkSources_899118_42.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/1058b3f3-333a-476a-86b8-975c3ceb9d9f_isrForCrosstalkSources_899118_42.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash ec5f58dce02b7be89627584325b6d431 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb434c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c8e1038e-041e-4d69-938e-2e8fe505157b_isrForCrosstalkSources_899118_45.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c8e1038e-041e-4d69-938e-2e8fe505157b_isrForCrosstalkSources_899118_45.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash fb39c803541af7b76fdb9add4e8beb9d parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/9121174e-03be-45f2-b48b-04bbdda4094b_isrForCrosstalkSources_899015_52.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/9121174e-03be-45f2-b48b-04bbdda4094b_isrForCrosstalkSources_899015_52.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash b73cf7b33ad6e9770d18605866cb5956 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e73f35f-1385-463b-b5ec-3fcaa6769f1e_isrForCrosstalkSources_899014_59.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e73f35f-1385-463b-b5ec-3fcaa6769f1e_isrForCrosstalkSources_899014_59.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 2320c3dbd99351f55ef3f01c64506f56 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0e29e968-6cde-41f6-b4c5-ec0e56eca151_isrForCrosstalkSources_899118_2.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0e29e968-6cde-41f6-b4c5-ec0e56eca151_isrForCrosstalkSources_899118_2.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 70e1336b775cb81867b55ff95f7a765f parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb432e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a3a93b53-e915-4709-9e41-50dc8045b897_isrForCrosstalkSources_899015_21.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a3a93b53-e915-4709-9e41-50dc8045b897_isrForCrosstalkSources_899015_21.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash aa6160bb9f860fb61d84b44fc8488459 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4c0620c8-1658-42a0-9821-cd473f736c03_isrForCrosstalkSources_899117_46.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4c0620c8-1658-42a0-9821-cd473f736c03_isrForCrosstalkSources_899117_46.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 2e097f8e2ec10bbc3525ad6b1f2fd181 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb42520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/01c2e98a-1706-4407-8d94-d6a25cf000e0_isrForCrosstalkSources_899118_36.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/01c2e98a-1706-4407-8d94-d6a25cf000e0_isrForCrosstalkSources_899118_36.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 6063405466054cf48249a2dde5379a33 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb800e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/dd3db005-cabc-471c-92ad-5d94ed5cb009_isrForCrosstalkSources_899014_24.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/dd3db005-cabc-471c-92ad-5d94ed5cb009_isrForCrosstalkSources_899014_24.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 4c021b03b89a3177e313521fc14b00b5 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/3247cf44-4311-46d6-a7b1-b26a6015da64_isrForCrosstalkSources_899119_19.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/3247cf44-4311-46d6-a7b1-b26a6015da64_isrForCrosstalkSources_899119_19.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 28140ddfa6363d71d6e366e93b1693c4 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb802c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8ae49a4f-bd6b-417e-b261-f2a3b7952d17_isrForCrosstalkSources_899118_37.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8ae49a4f-bd6b-417e-b261-f2a3b7952d17_isrForCrosstalkSources_899118_37.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 25b09683baf0a25e18389b3dee1f8989 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b570e4eb-97f6-414a-bb42-720cf75d46ae_isrForCrosstalkSources_899118_14.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b570e4eb-97f6-414a-bb42-720cf75d46ae_isrForCrosstalkSources_899118_14.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash c53b26207fd17479706eb9aca845a2a5 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c1804a3e-99ec-4917-b253-9e7264f1684d_isrForCrosstalkSources_899014_60.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c1804a3e-99ec-4917-b253-9e7264f1684d_isrForCrosstalkSources_899014_60.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash bb9349c8256d11aff6e3bdb7e62a73d0 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8516ddd-ca74-4354-b4a2-7905dad1965c_isrForCrosstalkSources_899014_23.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8516ddd-ca74-4354-b4a2-7905dad1965c_isrForCrosstalkSources_899014_23.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash c94c714f739056584c3a0ab03a76b52c parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb804a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7addd376-c9d3-4b78-81fa-8d4515557d61_isrForCrosstalkSources_899119_16.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7addd376-c9d3-4b78-81fa-8d4515557d61_isrForCrosstalkSources_899119_16.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 1184c184dac05c08eeb60f303ae280de parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb805e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca753064-9b9b-4441-9876-af88e0c21704_isrForCrosstalkSources_899016_33.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca753064-9b9b-4441-9876-af88e0c21704_isrForCrosstalkSources_899016_33.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 20fdf936117b8eae6936a010f9cfb36f parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/23bfcb45-c83e-4521-892d-0657f790903b_isrForCrosstalkSources_899014_12.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/23bfcb45-c83e-4521-892d-0657f790903b_isrForCrosstalkSources_899014_12.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 45031893ed6eb15f95e275649f38a776 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/bbe8312b-6c42-4674-90ae-26fd66663d7e_isrForCrosstalkSources_899015_32.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/bbe8312b-6c42-4674-90ae-26fd66663d7e_isrForCrosstalkSources_899015_32.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 1f7e2c7dc4b7a4faabff40343eb90db0 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6234f445-86f7-41de-9a77-d8cf1b61ca45_isrForCrosstalkSources_899119_34.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6234f445-86f7-41de-9a77-d8cf1b61ca45_isrForCrosstalkSources_899119_34.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 5c4f04d4c5d7cf8cceaa37f6582981b0 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb807c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc14b9d9-910d-4ee9-aba2-c7f536c689e8_isrForCrosstalkSources_899016_27.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc14b9d9-910d-4ee9-aba2-c7f536c689e8_isrForCrosstalkSources_899016_27.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash f8035f45b036a20142b79f59c65babcd parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/34aca2c0-f834-43c5-a7da-0667721eab2b_isrForCrosstalkSources_899014_6.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/34aca2c0-f834-43c5-a7da-0667721eab2b_isrForCrosstalkSources_899014_6.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash a2fe7a2afee91dc668c082e50fdd9dd2 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d28f833a-9e2b-4266-a690-5feff1a4ce0c_isrForCrosstalkSources_899118_60.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d28f833a-9e2b-4266-a690-5feff1a4ce0c_isrForCrosstalkSources_899118_60.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 679d679ed8848205bca96fca72c58118 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb809a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/250711dd-3188-48ce-a5f3-8ac467801301_isrForCrosstalkSources_899015_38.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/250711dd-3188-48ce-a5f3-8ac467801301_isrForCrosstalkSources_899015_38.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash ecc959a143c915b8cd54ce27d04287ba parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/9076866d-d3dc-425e-8fd4-e788cc19be6a_isrForCrosstalkSources_899016_1.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/9076866d-d3dc-425e-8fd4-e788cc19be6a_isrForCrosstalkSources_899016_1.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash cf568a32c914551153d9eb49e287392c parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/52f8c8b8-7c72-4683-9d15-9925109e4d51_isrForCrosstalkSources_899119_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/52f8c8b8-7c72-4683-9d15-9925109e4d51_isrForCrosstalkSources_899119_58.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 93d2e7aa5ee1548adaf141ac80de0c6e parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6fb35236-4754-46e9-a68f-a979eb9a921a_isrForCrosstalkSources_899119_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6fb35236-4754-46e9-a68f-a979eb9a921a_isrForCrosstalkSources_899119_2.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 78d1f85d2551716f4598cb28505ba8e5 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ef22e199-ab87-45c4-b12e-51d8752dfb24_isrForCrosstalkSources_899014_14.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ef22e199-ab87-45c4-b12e-51d8752dfb24_isrForCrosstalkSources_899014_14.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash eb02584da0d3fd9f83db282cf5c8fc2f parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6660b3de-6a8a-43ee-b75d-006003dfd9f7_isrForCrosstalkSources_899119_50.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6660b3de-6a8a-43ee-b75d-006003dfd9f7_isrForCrosstalkSources_899119_50.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 44f45878189c760395e3ee94574015ba parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/1f42ce4c-ada0-465c-b40c-a4666ec206a2_isrForCrosstalkSources_899117_13.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/1f42ce4c-ada0-465c-b40c-a4666ec206a2_isrForCrosstalkSources_899117_13.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 7074855f4389a464018a44cf8a91ce97 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/42f80eb3-0060-4977-a843-bf55f25f75cf_isrForCrosstalkSources_899117_31.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/42f80eb3-0060-4977-a843-bf55f25f75cf_isrForCrosstalkSources_899117_31.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 16574e4896053176ec4fe646fd25a862 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/293a7ce5-76f7-4ddb-9692-c5444c70e2b9_isrForCrosstalkSources_899016_5.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/293a7ce5-76f7-4ddb-9692-c5444c70e2b9_isrForCrosstalkSources_899016_5.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 613995e7bc3a40b825b1c56aaee18aef parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e25a4c19-1c65-453a-b2ef-99a2d244c799_isrForCrosstalkSources_899118_11.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e25a4c19-1c65-453a-b2ef-99a2d244c799_isrForCrosstalkSources_899118_11.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 70582aa950e60e4bcb03e273ca437ee3 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb80fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e6ed2489-93d0-472f-9d7f-bc8580a627f2_isrForCrosstalkSources_899117_23.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e6ed2489-93d0-472f-9d7f-bc8580a627f2_isrForCrosstalkSources_899117_23.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 487529b9f8db93bb5b431523821ade5a parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/78b01501-74de-43b8-a699-87b56d354bca_isrForCrosstalkSources_899015_17.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/78b01501-74de-43b8-a699-87b56d354bca_isrForCrosstalkSources_899015_17.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 677bd8179bdb20517c7bd7dc4046eaa3 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/18adbf30-c858-43a0-a632-6126bc1c2ca9_isrForCrosstalkSources_899016_9.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/18adbf30-c858-43a0-a632-6126bc1c2ca9_isrForCrosstalkSources_899016_9.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 378d85020b9ea1c2abf346d4028fcdf1 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f2d7670f-78bc-4c9c-b29e-737f0f38449e_isrForCrosstalkSources_899118_43.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f2d7670f-78bc-4c9c-b29e-737f0f38449e_isrForCrosstalkSources_899118_43.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 9167bdfdb2133a14fd058925dd417718 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb811c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2df80d40-058b-4e7c-8189-700607790dd3_isrForCrosstalkSources_899014_9.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2df80d40-058b-4e7c-8189-700607790dd3_isrForCrosstalkSources_899014_9.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 266a14ad326d2cb3bac7a29e5b5cb0e7 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/839e4a6f-c602-44e2-ac0f-a26fe1e2528e_isrForCrosstalkSources_899014_11.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/839e4a6f-c602-44e2-ac0f-a26fe1e2528e_isrForCrosstalkSources_899014_11.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash cfda87b0dc4b65d6c6311925a1d00ebf parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb813a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5a607f53-5ee0-4cfb-917b-08caa94e91d2_isrForCrosstalkSources_899118_61.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5a607f53-5ee0-4cfb-917b-08caa94e91d2_isrForCrosstalkSources_899118_61.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 8e5dd2f872e8792f0175d6fbc4ed1841 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4abbef5e-bb29-4069-b5b0-64ef47448e6b_isrForCrosstalkSources_899016_48.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4abbef5e-bb29-4069-b5b0-64ef47448e6b_isrForCrosstalkSources_899016_48.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash c2a257983bc9a8e69abf132d4afa2880 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a17f534f-4a62-4b50-9228-65a0d4478723_isrForCrosstalkSources_899119_51.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a17f534f-4a62-4b50-9228-65a0d4478723_isrForCrosstalkSources_899119_51.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 9a39688ef7b7498da95f36f5b9909259 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb814e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fd636dc2-9baf-491a-8b63-291e1d6068be_isrForCrosstalkSources_899014_26.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fd636dc2-9baf-491a-8b63-291e1d6068be_isrForCrosstalkSources_899014_26.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 09ec83cceb6300faccf94b6b4260ac4f parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3ec04df6-44f7-4ca5-9b75-8c3bdb0f8c9b_isrForCrosstalkSources_899014_46.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3ec04df6-44f7-4ca5-9b75-8c3bdb0f8c9b_isrForCrosstalkSources_899014_46.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 9775cb06dea15c2a5a9a6087731f1e62 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb816c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e62edc63-5a77-4d84-bdfb-697fbbcf1d7e_isrForCrosstalkSources_899117_37.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e62edc63-5a77-4d84-bdfb-697fbbcf1d7e_isrForCrosstalkSources_899117_37.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 056ac9f7c20b6a38fbc1257a053a2888 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8fc9aa2f-2040-4a93-8299-661d96f41d22_isrForCrosstalkSources_899015_51.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8fc9aa2f-2040-4a93-8299-661d96f41d22_isrForCrosstalkSources_899015_51.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash b437e60e1814591137508a0f208f6c5b parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb818a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4a182b7e-d712-42a7-a627-862009730ca1_isrForCrosstalkSources_899015_40.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4a182b7e-d712-42a7-a627-862009730ca1_isrForCrosstalkSources_899015_40.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 6ceda1fc3fbd37127d99c2cf82f2147d parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2c1cc094-26dc-4bbb-89a8-23dc59dd6c0d_isrForCrosstalkSources_899117_47.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2c1cc094-26dc-4bbb-89a8-23dc59dd6c0d_isrForCrosstalkSources_899117_47.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 5f4b2d64e5e5e901f502c8c48c1f1bd7 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5b579167-3049-429a-98d2-80ed54bcf50f_isrForCrosstalkSources_899119_6.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5b579167-3049-429a-98d2-80ed54bcf50f_isrForCrosstalkSources_899119_6.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 9a783dd579ffe06e3ce6b7af82a0f190 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb819e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f42a3965-43de-42f9-a7f6-ab6e8baf59f5_isrForCrosstalkSources_899016_46.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f42a3965-43de-42f9-a7f6-ab6e8baf59f5_isrForCrosstalkSources_899016_46.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 663eea81e61c86b3897b30a6a5e0c1f3 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/59acd42b-6d04-4847-8d61-1887d42f97b3_isrForCrosstalkSources_899015_58.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/59acd42b-6d04-4847-8d61-1887d42f97b3_isrForCrosstalkSources_899015_58.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash d5796b4bc0a0c1e8283d9fb2041849cc parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c0093db4-d24a-4c54-b755-b87dbac83736_isrForCrosstalkSources_899118_17.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c0093db4-d24a-4c54-b755-b87dbac83736_isrForCrosstalkSources_899118_17.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 7c1584e255a0c258c0998be9149c04cf parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abc10e17-e4f8-4c44-80d0-67c102403fbf_isrForCrosstalkSources_899015_22.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abc10e17-e4f8-4c44-80d0-67c102403fbf_isrForCrosstalkSources_899015_22.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 5610bf3a766adada6e328d57a818bca6 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e644f405-2955-4488-9baa-1c9f3ce39d94_isrForCrosstalkSources_899014_53.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e644f405-2955-4488-9baa-1c9f3ce39d94_isrForCrosstalkSources_899014_53.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 6ac7594a650e5397e6c3d83684f5606b parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8318b53a-ad6e-4ece-aaec-8f6534523380_isrForCrosstalkSources_899016_11.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8318b53a-ad6e-4ece-aaec-8f6534523380_isrForCrosstalkSources_899016_11.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash ecd74709033884834e1338dddc5dc9ed parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3835a85f-e485-43ef-bd1a-9895b2ccc28d_isrForCrosstalkSources_899016_4.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3835a85f-e485-43ef-bd1a-9895b2ccc28d_isrForCrosstalkSources_899016_4.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 871bc0f41a72c928b093bd1b0beb6603 parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0551946d-4bf6-4412-bbc9-1cf384bf8986_isrForCrosstalkSources_899119_35.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0551946d-4bf6-4412-bbc9-1cf384bf8986_isrForCrosstalkSources_899119_35.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 831e2080f4cb36efb120408585cfcf8c parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/969f3a09-de3d-42db-b44d-ec0cf4df1321_isrForCrosstalkSources_899016_54.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/969f3a09-de3d-42db-b44d-ec0cf4df1321_isrForCrosstalkSources_899016_54.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash d9580e0abf29a0056dd9954bfd9bfedf parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb81f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/890deb1e-de43-4ecd-bdb0-9afe9c6f3b9b_isrForCrosstalkSources_899117_8.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/890deb1e-de43-4ecd-bdb0-9afe9c6f3b9b_isrForCrosstalkSources_899117_8.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash c8fa5f3225937a22a6ed67f919ed50ae parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5098cfd4-7c30-446f-849e-c11eab41769d_isrForCrosstalkSources_899117_34.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5098cfd4-7c30-446f-849e-c11eab41769d_isrForCrosstalkSources_899117_34.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash d82fef2cc69cc9375f726874d4d4812c parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb820c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8f714d4d-bce0-417b-b79e-6747e3a1662d_isrForCrosstalkSources_899118_25.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8f714d4d-bce0-417b-b79e-6747e3a1662d_isrForCrosstalkSources_899118_25.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 2854bdd784c05fda11d4ad42ee4f98f9 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5e94a4bb-8a0a-43ee-965d-18f790b46aa1_isrForCrosstalkSources_899118_8.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5e94a4bb-8a0a-43ee-965d-18f790b46aa1_isrForCrosstalkSources_899118_8.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash ef45da2d69f2ba5c0197278311c8836c parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f20a3648-5a91-4607-8a25-ec02379701ba_isrForCrosstalkSources_899118_6.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f20a3648-5a91-4607-8a25-ec02379701ba_isrForCrosstalkSources_899118_6.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash d6014d5dfd039007dc85ebad50e3c5d5 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb822a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5cb79f8d-f338-4394-ac39-5b4fec85cd7a_isrForCrosstalkSources_899014_3.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5cb79f8d-f338-4394-ac39-5b4fec85cd7a_isrForCrosstalkSources_899014_3.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 53e25e7b40d3e4569a8b96ef6033208d parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/097657a1-6d04-4c89-8873-e04fd2b1b5ea_isrForCrosstalkSources_899117_17.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/097657a1-6d04-4c89-8873-e04fd2b1b5ea_isrForCrosstalkSources_899117_17.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 969f13bbae10112b7e19a18bde3450e7 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb823e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3559d182-aa6a-4d21-8d64-c49de30c112a_isrForCrosstalkSources_899016_55.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3559d182-aa6a-4d21-8d64-c49de30c112a_isrForCrosstalkSources_899016_55.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash fe2da04dbd6a5be049e45908663ae127 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5bc26814-288f-4eae-861a-a5c4a8aa24cd_isrForCrosstalkSources_899014_28.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5bc26814-288f-4eae-861a-a5c4a8aa24cd_isrForCrosstalkSources_899014_28.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 3e00b53830eba4193280420dfaefa1d8 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d815734c-64a8-4ca7-ad4f-b2a3f458ece5_isrForCrosstalkSources_899119_41.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d815734c-64a8-4ca7-ad4f-b2a3f458ece5_isrForCrosstalkSources_899119_41.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash ec846b098fca0ca003165d3b4943769b parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb825c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/75db9ec5-898a-4b28-8e07-72b57b25c8fa_isrForCrosstalkSources_899015_5.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/75db9ec5-898a-4b28-8e07-72b57b25c8fa_isrForCrosstalkSources_899015_5.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 9c0cb2a957de571ea46147f286ccc3ac parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b2d05f5c-f140-4371-8cce-80a765194ea5_isrForCrosstalkSources_899119_57.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b2d05f5c-f140-4371-8cce-80a765194ea5_isrForCrosstalkSources_899119_57.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 1f0eb559afde2707543fb5cd95d37fe6 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b40a0a1d-5a16-4d3d-94af-2bbcdcaac047_isrForCrosstalkSources_899014_5.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b40a0a1d-5a16-4d3d-94af-2bbcdcaac047_isrForCrosstalkSources_899014_5.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 382edfa821ebe8a7b90480bd14f6c371 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c70339d0-771c-4c41-bf54-caf08de7b822_isrForCrosstalkSources_899118_54.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c70339d0-771c-4c41-bf54-caf08de7b822_isrForCrosstalkSources_899118_54.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash f792f3374bae4b7a2503baf483b75110 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb827a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8d282b43-06c9-4dfd-8c82-68c327832a56_isrForCrosstalkSources_899014_62.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8d282b43-06c9-4dfd-8c82-68c327832a56_isrForCrosstalkSources_899014_62.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 7a75c8c5f7a6794fbd6715ba70b29410 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb828e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/333375cc-4b8e-43b1-a070-dbdddb4154cd_isrForCrosstalkSources_899016_26.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/333375cc-4b8e-43b1-a070-dbdddb4154cd_isrForCrosstalkSources_899016_26.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 30b586d4a35630d42b5acd92a3cdd9cd parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0dcf3bb3-1f21-49d2-b8d5-03d41af03bd2_isrForCrosstalkSources_899118_46.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0dcf3bb3-1f21-49d2-b8d5-03d41af03bd2_isrForCrosstalkSources_899118_46.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 009e799693220fc90b46a39dfce7bb1b parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/94bb526a-6759-4085-b54e-695eee2ce630_isrForCrosstalkSources_899015_7.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/94bb526a-6759-4085-b54e-695eee2ce630_isrForCrosstalkSources_899015_7.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 3347fef04a2b2089e17f4da6964d64c8 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/73bed8b5-0b94-4199-b4d4-157b0b6ffd9a_isrForCrosstalkSources_899118_10.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/73bed8b5-0b94-4199-b4d4-157b0b6ffd9a_isrForCrosstalkSources_899118_10.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 97a17d5ca7c6a087ad96c99fdc430c83 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/de0fc91e-7696-43b5-9788-09a1f774c4a9_isrForCrosstalkSources_899015_25.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/de0fc91e-7696-43b5-9788-09a1f774c4a9_isrForCrosstalkSources_899015_25.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 04f52cdcfaf3eb768cf8ca6947bb2604 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e129d99f-fef4-4a55-8d8e-8808ee1e0faa_isrForCrosstalkSources_899014_10.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e129d99f-fef4-4a55-8d8e-8808ee1e0faa_isrForCrosstalkSources_899014_10.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash e01f2df3f7291af4d7ca928c056e7972 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/25cc433f-0c74-4311-96c1-b8b65c8ba1fa_isrForCrosstalkSources_899117_42.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/25cc433f-0c74-4311-96c1-b8b65c8ba1fa_isrForCrosstalkSources_899117_42.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 7b3eadb38223804b3059a2039bb78646 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1feabea-42fd-40e6-ad51-0cd4f4e93b38_isrForCrosstalkSources_899015_28.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1feabea-42fd-40e6-ad51-0cd4f4e93b38_isrForCrosstalkSources_899015_28.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash b583f58396bd69fd28b564ad09400651 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/359425fd-5b77-46bf-9532-f77badf8ac8e_isrForCrosstalkSources_899119_52.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/359425fd-5b77-46bf-9532-f77badf8ac8e_isrForCrosstalkSources_899119_52.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash d7865d80bda2e89984641b7d354eba9d parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/dfaacbd7-eeb9-4e38-87aa-b03b5598189e_isrForCrosstalkSources_899119_60.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/dfaacbd7-eeb9-4e38-87aa-b03b5598189e_isrForCrosstalkSources_899119_60.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 175585b2a95c6c8b77c8fbbb271eed20 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0544773c-6a20-44ca-8e96-c4d850980448_isrForCrosstalkSources_899015_45.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0544773c-6a20-44ca-8e96-c4d850980448_isrForCrosstalkSources_899015_45.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 259c1ca2195130e6a0a6e6b427dd96c9 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb82fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5dbdee85-47ee-4f13-8623-e9e8e8b1e1eb_isrForCrosstalkSources_899014_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5dbdee85-47ee-4f13-8623-e9e8e8b1e1eb_isrForCrosstalkSources_899014_2.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 26eaee8f20cfd8470f433fddb893ed6f parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ce955da6-04d2-4492-b5bb-b16d01411dd6_isrForCrosstalkSources_899117_41.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ce955da6-04d2-4492-b5bb-b16d01411dd6_isrForCrosstalkSources_899117_41.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 6f7e08f34be2923e370092c37cf1960f parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb831a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/45dd7ed4-e6a5-4b98-9b98-a0dc5a8425d0_isrForCrosstalkSources_899015_48.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/45dd7ed4-e6a5-4b98-9b98-a0dc5a8425d0_isrForCrosstalkSources_899015_48.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash f8b935373252ba4efae3b08274045a35 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/23703ab6-28c7-42fa-b6ed-23e6afe3facc_isrForCrosstalkSources_899119_36.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/23703ab6-28c7-42fa-b6ed-23e6afe3facc_isrForCrosstalkSources_899119_36.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 9db3c4d6e2bbac2034e16c2750fb75a1 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fbb290aa-9f6c-4a95-837e-d9acaaad2dcd_isrForCrosstalkSources_899014_61.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fbb290aa-9f6c-4a95-837e-d9acaaad2dcd_isrForCrosstalkSources_899014_61.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 678d83562059ec94b0c4c09679453a9a parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb832e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4f80cb53-de45-4b97-8a00-36121d6475e9_isrForCrosstalkSources_899016_43.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4f80cb53-de45-4b97-8a00-36121d6475e9_isrForCrosstalkSources_899016_43.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 59b46f2b901a56afaff58d12446e6dfb parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f50a7519-1692-4bd3-9a62-5ad8323243e2_isrForCrosstalkSources_899014_56.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f50a7519-1692-4bd3-9a62-5ad8323243e2_isrForCrosstalkSources_899014_56.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash df9e45fa8d84edab99e4e986ba559dac parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb834c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dc802ae1-a9d5-4ee0-8f1b-d3c57776e38b_isrForCrosstalkSources_899118_15.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dc802ae1-a9d5-4ee0-8f1b-d3c57776e38b_isrForCrosstalkSources_899118_15.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 6ffd4b205fc4ad6b16c2870f0cf4371e parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 13:18:56 proc_lsst.multi:146 [INFO] found job 86094 in provider local proc_lsst.multi INFO: found job 86094 in provider local parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ecf977dd-9e22-4e0d-9582-7e384a6da226_isrForCrosstalkSources_899014_22.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ecf977dd-9e22-4e0d-9582-7e384a6da226_isrForCrosstalkSources_899014_22.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 2b5ff0f2d6db720deed941a3e134dde0 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb43240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/5851a6da-120c-4c8a-9ec4-ec831d547667_isrForCrosstalkSources_899016_22.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/5851a6da-120c-4c8a-9ec4-ec831d547667_isrForCrosstalkSources_899016_22.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash b03d9a95346c6de2c70c72c54a4ad968 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f72d04d4-504b-46a1-90d7-a9b83872958c_isrForCrosstalkSources_899015_29.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f72d04d4-504b-46a1-90d7-a9b83872958c_isrForCrosstalkSources_899015_29.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 00314bc404bfb4c1353a49f006d3bb4e parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f6b7ae32-71d4-4616-bdf8-955b9d1991e5_isrForCrosstalkSources_899016_60.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f6b7ae32-71d4-4616-bdf8-955b9d1991e5_isrForCrosstalkSources_899016_60.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 560d8b62552ce5e94ea1aac374126bbd parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb839c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5185c45b-0106-428b-8a65-be342f4c1e78_isrForCrosstalkSources_899119_49.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5185c45b-0106-428b-8a65-be342f4c1e78_isrForCrosstalkSources_899119_49.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 697f107138595b64e75e53c478b4035a parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b94b1871-d735-4f24-bd0f-e2d09ec12999_isrForCrosstalkSources_899118_38.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b94b1871-d735-4f24-bd0f-e2d09ec12999_isrForCrosstalkSources_899118_38.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 15742aa5f50b06b6e1bb08ba6c46a651 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b84f07d-c8ff-4f67-a387-5579bbb9af9d_isrForCrosstalkSources_899015_50.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b84f07d-c8ff-4f67-a387-5579bbb9af9d_isrForCrosstalkSources_899015_50.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash e6ed6f303b1b792464f9a37f81f4f144 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5d82899e-e277-402b-8525-e8a657b9218b_isrForCrosstalkSources_899117_26.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5d82899e-e277-402b-8525-e8a657b9218b_isrForCrosstalkSources_899117_26.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 686b8c84bdd50fe74fb4ead484696147 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 153 parsl.dataflow.strategy DEBUG: Executor multi has 153 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 38 more blocks parsl.executors.status_handling INFO: Scaling out by 38 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-13 13:18:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-13 13:18:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 13:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b86bd2d9-961c-402c-851d-b3a61e288975_isrForCrosstalkSources_899118_4.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b86bd2d9-961c-402c-851d-b3a61e288975_isrForCrosstalkSources_899118_4.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash c25a03d613311ccde608ccbefb867ee7 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/776cbde3-8206-4636-8b15-8dfe62a5a942_isrForCrosstalkSources_899119_7.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/776cbde3-8206-4636-8b15-8dfe62a5a942_isrForCrosstalkSources_899119_7.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 68959eea53663cbfa5bdc47f70e5b569 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/47f113c0-ec57-49f1-af68-81444d457379_isrForCrosstalkSources_899119_59.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/47f113c0-ec57-49f1-af68-81444d457379_isrForCrosstalkSources_899119_59.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 20d21880b590520294098857c8d41631 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d1079139-28a3-4a78-ac0b-da56d759ff25_isrForCrosstalkSources_899117_62.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d1079139-28a3-4a78-ac0b-da56d759ff25_isrForCrosstalkSources_899117_62.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 650f60ad3be3a74b20449a198989b5a8 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6f3274e0-6d48-4665-9d9a-30e6639c17dc_isrForCrosstalkSources_899119_43.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6f3274e0-6d48-4665-9d9a-30e6639c17dc_isrForCrosstalkSources_899119_43.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash bbb1f3b9bf76e61daf551c9d71e640ab parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b924148b-3e65-4ab3-a15b-f3ebb1d98560_isrForCrosstalkSources_899118_1.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b924148b-3e65-4ab3-a15b-f3ebb1d98560_isrForCrosstalkSources_899118_1.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 11252d614262e570b4663f0b241ec190 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb837e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b43c01f9-4d69-42d1-81d9-4937bcc73c78_isrForCrosstalkSources_899016_16.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b43c01f9-4d69-42d1-81d9-4937bcc73c78_isrForCrosstalkSources_899016_16.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash da8b19024b034e3233392a0f0998aff7 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb836a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a1c80bbb-0399-4093-8ecd-499c245fa66d_isrForCrosstalkSources_899016_50.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a1c80bbb-0399-4093-8ecd-499c245fa66d_isrForCrosstalkSources_899016_50.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 0183a77f543e8168cfc0a93074a188bf parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ec7ab53f-3c49-4090-8624-b2a664a43e7b_isrForCrosstalkSources_899118_41.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ec7ab53f-3c49-4090-8624-b2a664a43e7b_isrForCrosstalkSources_899118_41.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash a5e35bbb5f8443e9a7bec719744dcfb7 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/858eff25-409c-4466-8211-0601acf237eb_isrForCrosstalkSources_899119_37.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/858eff25-409c-4466-8211-0601acf237eb_isrForCrosstalkSources_899119_37.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 86265aa5e7002144e649ba92dada8982 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f3f836db-3aea-40d4-8daf-ab25ce02145f_isrForCrosstalkSources_899118_32.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f3f836db-3aea-40d4-8daf-ab25ce02145f_isrForCrosstalkSources_899118_32.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 98e65414eddd2fafa7bed7171bdeb911 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/505b8fb0-22c9-42ec-945a-9edd258613d2_isrForCrosstalkSources_899015_46.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/505b8fb0-22c9-42ec-945a-9edd258613d2_isrForCrosstalkSources_899015_46.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 5e82c72818605a4a43b13f76cf9ca230 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b0b482da-3d4f-484e-928b-00c63e9478b0_isrForCrosstalkSources_899014_47.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b0b482da-3d4f-484e-928b-00c63e9478b0_isrForCrosstalkSources_899014_47.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 74828b32565023748f0ce859dd6b24a4 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e0289fc9-3476-4d63-9861-da0ae7a35ffc_isrForCrosstalkSources_899119_10.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e0289fc9-3476-4d63-9861-da0ae7a35ffc_isrForCrosstalkSources_899119_10.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash e50dd0a6a33c49dd51c1e6fed3f83b75 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c43a9bc-2c32-43ce-ac5b-0207fb512cb6_isrForCrosstalkSources_899117_22.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c43a9bc-2c32-43ce-ac5b-0207fb512cb6_isrForCrosstalkSources_899117_22.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 405d5e4a5ece4f875449e849c8fc3f31 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/203f1d9d-8498-4775-81ae-85ccbbcfe7aa_isrForCrosstalkSources_899119_47.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/203f1d9d-8498-4775-81ae-85ccbbcfe7aa_isrForCrosstalkSources_899119_47.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash a75b295006816f47a6f5e86cbc3adcde parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3e83f924-6b1b-4e76-902d-3ab22ff1cf9d_isrForCrosstalkSources_899014_13.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3e83f924-6b1b-4e76-902d-3ab22ff1cf9d_isrForCrosstalkSources_899014_13.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash f7a3768ce4467f7b8617c1578e8c499f parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f92b8999-4b57-47da-897b-0702d9ff3450_isrForCrosstalkSources_899118_58.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f92b8999-4b57-47da-897b-0702d9ff3450_isrForCrosstalkSources_899118_58.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash ebf3f7b9731b7049b7bad5df73d255bf parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/79472058-007e-4271-9798-dd21dae467c4_isrForCrosstalkSources_899119_29.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/79472058-007e-4271-9798-dd21dae467c4_isrForCrosstalkSources_899119_29.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 7bf59d0639b285833931c2f650ffdf41 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/58e00699-000e-4f88-ab0d-0e9c221e1715_isrForCrosstalkSources_899016_38.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/58e00699-000e-4f88-ab0d-0e9c221e1715_isrForCrosstalkSources_899016_38.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 8844d99fcf62a7d21d409e55497992e6 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d85f8ce8-3a5a-4059-bf83-be4af5c25071_isrForCrosstalkSources_899118_44.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d85f8ce8-3a5a-4059-bf83-be4af5c25071_isrForCrosstalkSources_899118_44.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 1b0dfa97e29e8efee2f87e7d3439975e parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/3f10c7e8-e566-48d1-8e41-e6da5a922cda_isrForCrosstalkSources_899118_47.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/3f10c7e8-e566-48d1-8e41-e6da5a922cda_isrForCrosstalkSources_899118_47.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash f6cb296b8f0a31b9b6a8902842abd50a parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e4f9ba3b-2c9e-4657-8996-c486f2c39885_isrForCrosstalkSources_899118_20.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e4f9ba3b-2c9e-4657-8996-c486f2c39885_isrForCrosstalkSources_899118_20.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 9bc6191bb25c8917649946b93af5d79b parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/da55fe65-fc5d-4f8d-9469-9a72ad0ccedd_isrForCrosstalkSources_899117_58.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/da55fe65-fc5d-4f8d-9469-9a72ad0ccedd_isrForCrosstalkSources_899117_58.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 694768aa2f154011a6fda44ab542adef parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ab6b987e-1523-40c2-b6b7-16b2eb781d41_isrForCrosstalkSources_899014_16.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ab6b987e-1523-40c2-b6b7-16b2eb781d41_isrForCrosstalkSources_899014_16.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 3f7f8522560f0b24c5933d37419ea47e parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a9ec1647-204b-448c-9115-18efbe2e26f7_isrForCrosstalkSources_899119_27.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a9ec1647-204b-448c-9115-18efbe2e26f7_isrForCrosstalkSources_899119_27.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash b5ce796fbb484ffbcf8875783ac02138 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4c171e3b-b35b-4fb9-8352-5e4a82a135a8_isrForCrosstalkSources_899119_28.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4c171e3b-b35b-4fb9-8352-5e4a82a135a8_isrForCrosstalkSources_899119_28.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 3fef690f7c03df22faa55ae7b606ed0f parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/331acef6-77b2-4ead-b33c-0cd2873a3615_isrForCrosstalkSources_899117_52.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/331acef6-77b2-4ead-b33c-0cd2873a3615_isrForCrosstalkSources_899117_52.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 1d2fa7d3440ab6dfc215f1f33fb1d979 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f5fba590-0967-435b-b1a1-43eb841ae07c_isrForCrosstalkSources_899016_6.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f5fba590-0967-435b-b1a1-43eb841ae07c_isrForCrosstalkSources_899016_6.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 4c9c79b9dc29612b34224aca67ac7794 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0760576d-7984-4a2e-8d29-9094b1430222_isrForCrosstalkSources_899119_32.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0760576d-7984-4a2e-8d29-9094b1430222_isrForCrosstalkSources_899119_32.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash a6ebbe7d83b22346a911d7a8a9ca33a7 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/29c782ba-be5e-4bb1-b95e-8866e3da361b_isrForCrosstalkSources_899118_21.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/29c782ba-be5e-4bb1-b95e-8866e3da361b_isrForCrosstalkSources_899118_21.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 209eacf7db3965391d0b8b6150421c9a parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/777df358-7ba0-48ed-a924-4cd0449a242e_isrForCrosstalkSources_899117_54.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/777df358-7ba0-48ed-a924-4cd0449a242e_isrForCrosstalkSources_899117_54.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash efb3e0aed1bf60a895c80c51d678e64b parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0d808d00-5aa8-4482-89c0-149bbc6487a3_isrForCrosstalkSources_899117_57.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0d808d00-5aa8-4482-89c0-149bbc6487a3_isrForCrosstalkSources_899117_57.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 086ed435aa3b962adeda8a398b592c0c parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 13:18:56 proc_lsst.multi:170 [INFO] [multi] job_id 17014789 proc_lsst.multi INFO: [multi] job_id 17014789 2024-03-13 13:18:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014789 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014789 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17014789 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-13 13:18:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-13 13:18:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abf779ac-ee35-438c-b1d7-98df252e672b_isrForCrosstalkSources_899015_16.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abf779ac-ee35-438c-b1d7-98df252e672b_isrForCrosstalkSources_899015_16.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash f3e26cb1b34050a07268e7d89964cce4 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/39642c65-d243-4fff-bb3f-8e837dd7d3ba_isrForCrosstalkSources_899015_57.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/39642c65-d243-4fff-bb3f-8e837dd7d3ba_isrForCrosstalkSources_899015_57.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash e6d2373ac89def5768c2c2950c7d68a7 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f739003f-ea83-49a3-a2c1-a84233f9266f_isrForCrosstalkSources_899016_18.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f739003f-ea83-49a3-a2c1-a84233f9266f_isrForCrosstalkSources_899016_18.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash a333f1683a156e86886973b982b7629a parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c3b782e1-1cb9-4870-9090-9a482070e816_isrForCrosstalkSources_899014_57.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c3b782e1-1cb9-4870-9090-9a482070e816_isrForCrosstalkSources_899014_57.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 2aad21f83a40c24b8518b3be4666a644 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1bac2cc1-1329-4768-b414-55e16a6cf880_isrForCrosstalkSources_899119_17.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1bac2cc1-1329-4768-b414-55e16a6cf880_isrForCrosstalkSources_899119_17.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash cc9d974fa7ff4f2873827b04ed1aca72 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f1cbbbc-a15a-45e5-ad77-ef47ffeb9480_isrForCrosstalkSources_899015_43.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f1cbbbc-a15a-45e5-ad77-ef47ffeb9480_isrForCrosstalkSources_899015_43.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash e3a2d8a742421c0619abd4bb91b0e371 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/96b07a8d-c208-4c39-853c-f812e90339e6_isrForCrosstalkSources_899119_38.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/96b07a8d-c208-4c39-853c-f812e90339e6_isrForCrosstalkSources_899119_38.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash b780d3e26248307db9a7d76d14a8d0a0 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/dfa67f77-769d-4811-b7aa-bccf715feb25_isrForCrosstalkSources_899015_14.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/dfa67f77-769d-4811-b7aa-bccf715feb25_isrForCrosstalkSources_899015_14.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash ae249e37f67729676e3f566c33767d58 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8c9f03e-3da6-47d5-a6fb-1ba9fe0e9d1c_isrForCrosstalkSources_899014_41.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8c9f03e-3da6-47d5-a6fb-1ba9fe0e9d1c_isrForCrosstalkSources_899014_41.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash e870a1ed5242cfd4e025652aa2f6a11e parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/2dcde062-9304-411a-966b-d56268235327_isrForCrosstalkSources_899119_13.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/2dcde062-9304-411a-966b-d56268235327_isrForCrosstalkSources_899119_13.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 43adbf0a2c539ea59f98506ecffea119 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7ee9be3f-e449-434f-b906-ad47a470d171_isrForCrosstalkSources_899015_44.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7ee9be3f-e449-434f-b906-ad47a470d171_isrForCrosstalkSources_899015_44.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 2a85a38c020ef84d76863bab75ec74cc parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bf44e037-8286-45a2-bf62-899ef0adfed7_isrForCrosstalkSources_899118_39.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bf44e037-8286-45a2-bf62-899ef0adfed7_isrForCrosstalkSources_899118_39.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 14105a262146c79be229f2f658767a19 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/906c27af-81bf-4843-9936-a44d81c0d9e8_isrForCrosstalkSources_899117_25.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/906c27af-81bf-4843-9936-a44d81c0d9e8_isrForCrosstalkSources_899117_25.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 60a3609371a3db4441f38c836396c075 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2ff13a12-92fb-4bbb-94a8-a81f67520e41_isrForCrosstalkSources_899015_20.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2ff13a12-92fb-4bbb-94a8-a81f67520e41_isrForCrosstalkSources_899015_20.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 7038173481ea95048faa85f376890df7 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2ca1d32a-1104-4432-ba04-51f660c4ca8d_isrForCrosstalkSources_899117_55.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2ca1d32a-1104-4432-ba04-51f660c4ca8d_isrForCrosstalkSources_899117_55.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 09b5b10af9040d5f849ece3a4b09c2d6 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/923ffaaa-9c41-434f-95a5-66edb1b7618e_isrForCrosstalkSources_899117_19.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/923ffaaa-9c41-434f-95a5-66edb1b7618e_isrForCrosstalkSources_899117_19.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 9cde7b6fb5a8222fe7d010881229b2ca parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85a0662-cbeb-495a-a299-05913895491e_isrForCrosstalkSources_899016_3.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85a0662-cbeb-495a-a299-05913895491e_isrForCrosstalkSources_899016_3.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash d992e113b82fefce8bd384dcbd3cb830 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c836e4cb-630a-4d32-b190-373221fd7a7d_isrForCrosstalkSources_899119_55.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c836e4cb-630a-4d32-b190-373221fd7a7d_isrForCrosstalkSources_899119_55.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 45c7370a5c967ddbeb2280e782a8b90b parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/202a6d61-1bf9-47b0-aa68-e8bd605ceebc_isrForCrosstalkSources_899119_23.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/202a6d61-1bf9-47b0-aa68-e8bd605ceebc_isrForCrosstalkSources_899119_23.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 2b4814f8e9ddd41a2401d561b5d81739 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7c8394b5-8047-485d-947f-290cb5c36953_isrForCrosstalkSources_899016_51.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7c8394b5-8047-485d-947f-290cb5c36953_isrForCrosstalkSources_899016_51.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 7a4bb5a00572489b07ddc2996a6bf4d1 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28418052-5fa8-4f43-af31-4a46100eda3e_isrForCrosstalkSources_899014_19.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28418052-5fa8-4f43-af31-4a46100eda3e_isrForCrosstalkSources_899014_19.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 3e78ea05ad7dda3675c4ac832a870cf5 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/047b91ee-b310-483a-8380-0c75708b5a91_isrForCrosstalkSources_899119_21.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/047b91ee-b310-483a-8380-0c75708b5a91_isrForCrosstalkSources_899119_21.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash a710fa22dab97ad839da34d359aa07a0 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05fd8a2d-b3e8-46e6-83c1-11e88beb0d32_isrForCrosstalkSources_899117_32.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05fd8a2d-b3e8-46e6-83c1-11e88beb0d32_isrForCrosstalkSources_899117_32.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 7898199db8bfe832b611ef0269b22033 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d390164b-3725-4b6b-9212-cc5ac70d50d6_isrForCrosstalkSources_899118_56.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d390164b-3725-4b6b-9212-cc5ac70d50d6_isrForCrosstalkSources_899118_56.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 6fc75b49b3804c09e90661cde812956c parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/90046cae-8c68-4579-aa10-7b5d238ef888_isrForCrosstalkSources_899016_19.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/90046cae-8c68-4579-aa10-7b5d238ef888_isrForCrosstalkSources_899016_19.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 51b173d1da5c85a2ec485b4ab5bd3ea7 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f305ee7e-ee23-416c-a8e4-352b167fd8dc_isrForCrosstalkSources_899015_41.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f305ee7e-ee23-416c-a8e4-352b167fd8dc_isrForCrosstalkSources_899015_41.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash ed7713952cca51f62a802f918383db58 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77e9bb1c-c31e-438c-a2f0-2e282290b010_isrForCrosstalkSources_899117_53.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77e9bb1c-c31e-438c-a2f0-2e282290b010_isrForCrosstalkSources_899117_53.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 7980e818c5380e0efa39749098ccee01 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/de7a9fcc-d35e-4051-b62d-9bd023be19ce_isrForCrosstalkSources_899014_20.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/de7a9fcc-d35e-4051-b62d-9bd023be19ce_isrForCrosstalkSources_899014_20.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 1c69d1977126104f5ae16871faba7525 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/60998f79-7f78-4b77-a94e-63a5ae99c8c0_isrForCrosstalkSources_899118_35.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/60998f79-7f78-4b77-a94e-63a5ae99c8c0_isrForCrosstalkSources_899118_35.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash b15e4995307c4a615e5afd23563bfb4e parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/221f32ff-3099-4422-a643-727550686b7d_isrForCrosstalkSources_899118_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/221f32ff-3099-4422-a643-727550686b7d_isrForCrosstalkSources_899118_48.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash fe743bb299565eb9269091e3a05eadde parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c6d3b75b-0033-4e13-b666-63f32136dbd4_isrForCrosstalkSources_899014_54.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c6d3b75b-0033-4e13-b666-63f32136dbd4_isrForCrosstalkSources_899014_54.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash d10429b7324c7f4662749b81c9d45f85 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7db8b61c-187c-4f53-9046-90f16349b1ea_isrForCrosstalkSources_899117_11.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7db8b61c-187c-4f53-9046-90f16349b1ea_isrForCrosstalkSources_899117_11.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 4b30c50606f6ea0fff619ecaaad46851 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/566c1e18-52a3-418a-ad42-fd6136b62845_isrForCrosstalkSources_899118_18.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/566c1e18-52a3-418a-ad42-fd6136b62845_isrForCrosstalkSources_899118_18.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 4de2bc39b9c0e1e5dcb87a4199899b80 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3087576b-ea8e-4f7c-a93a-5940d6ad48c3_isrForCrosstalkSources_899117_50.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3087576b-ea8e-4f7c-a93a-5940d6ad48c3_isrForCrosstalkSources_899117_50.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 5b81a35860cb235dbade8288e21905da parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0e162909-f29e-4026-b8f6-8d08f821b37e_isrForCrosstalkSources_899119_15.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0e162909-f29e-4026-b8f6-8d08f821b37e_isrForCrosstalkSources_899119_15.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 1845cbdb68be10024cf9bd1f07420734 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dd558f04-c20d-4cdb-a0a8-dba1a6a45da7_isrForCrosstalkSources_899118_19.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dd558f04-c20d-4cdb-a0a8-dba1a6a45da7_isrForCrosstalkSources_899118_19.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 7d48821c13b9ba2f7085cfc12a60cf13 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c1c027de-bd77-41b2-b2bc-1cbdd8cfaef8_isrForCrosstalkSources_899015_12.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c1c027de-bd77-41b2-b2bc-1cbdd8cfaef8_isrForCrosstalkSources_899015_12.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash d146ee24ea5a0efe4654a2491e1a9d66 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/973bd1c8-03cf-4dab-896b-dc7e714dbf43_isrForCrosstalkSources_899119_62.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/973bd1c8-03cf-4dab-896b-dc7e714dbf43_isrForCrosstalkSources_899119_62.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 6b3561fa12f98cd7205043ac3ac733ba parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fe5da567-c815-4bb6-af2e-9dbe1cd7bed2_isrForCrosstalkSources_899118_57.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fe5da567-c815-4bb6-af2e-9dbe1cd7bed2_isrForCrosstalkSources_899118_57.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 4aac257a9189f23ae3e1fee893a2565b parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fd743f40-fa09-4310-b600-2480100f9406_isrForCrosstalkSources_899016_45.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fd743f40-fa09-4310-b600-2480100f9406_isrForCrosstalkSources_899016_45.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 9239611b4c53d08f3cd4402aa78a4755 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/166d73c3-fb5a-477a-a350-4a9c0d2c88fc_isrForCrosstalkSources_899016_53.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/166d73c3-fb5a-477a-a350-4a9c0d2c88fc_isrForCrosstalkSources_899016_53.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 1304e0def7391e87ed9c0a1e0c36a3b4 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5cbe0414-6ccd-41d0-a090-4043bfbc0647_isrForCrosstalkSources_899119_26.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5cbe0414-6ccd-41d0-a090-4043bfbc0647_isrForCrosstalkSources_899119_26.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 58edbdd1913767b4f86df38b2f91a172 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb63602d-909f-4ca2-b0c2-c6f5531d8799_isrForCrosstalkSources_899016_30.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb63602d-909f-4ca2-b0c2-c6f5531d8799_isrForCrosstalkSources_899016_30.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash a1f68bb26bae4b80fc4da5da273a5d84 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16cf4b91-083e-4e72-bfba-b881f2a20156_isrForCrosstalkSources_899117_39.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16cf4b91-083e-4e72-bfba-b881f2a20156_isrForCrosstalkSources_899117_39.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 0f7be5a81e62c583a965b11991f6b233 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/7634d1d8-e451-4876-8333-15a7eee51922_isrForCrosstalkSources_899118_31.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/7634d1d8-e451-4876-8333-15a7eee51922_isrForCrosstalkSources_899118_31.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash ee4ceb80c413cfb176ede5de24ea6f86 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/1e30a0a0-5aea-4532-b78b-6554b021eb58_isrForCrosstalkSources_899015_30.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/1e30a0a0-5aea-4532-b78b-6554b021eb58_isrForCrosstalkSources_899015_30.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 71f91129fa5d05f11ba703c102dd5061 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 13:18:57 proc_lsst.multi:170 [INFO] [multi] job_id 17014790 proc_lsst.multi INFO: [multi] job_id 17014790 2024-03-13 13:18:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 13:18:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014790 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014790 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17014790 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-13 13:18:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-13 13:18:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 13:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f494afb4-f0da-4912-9e1f-eef1fdc2fc25_isrForCrosstalkSources_899014_18.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f494afb4-f0da-4912-9e1f-eef1fdc2fc25_isrForCrosstalkSources_899014_18.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 5cec316ae9e7e143c1f60eae029185f0 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5453c8f4-fabd-40a3-a81d-55a132ac57c8_isrForCrosstalkSources_899118_34.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5453c8f4-fabd-40a3-a81d-55a132ac57c8_isrForCrosstalkSources_899118_34.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash d5e626904fa96155e2851e9489ec5b00 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5e57e2b3-f845-4601-b012-8ebc427eecd6_isrForCrosstalkSources_899015_56.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5e57e2b3-f845-4601-b012-8ebc427eecd6_isrForCrosstalkSources_899015_56.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 216e7cc8d69287a45ec0f5ac3234a0b3 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8d3a879d-f8aa-4061-8e6a-14930e5149cc_isrForCrosstalkSources_899119_25.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8d3a879d-f8aa-4061-8e6a-14930e5149cc_isrForCrosstalkSources_899119_25.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 7f6a1f4f2a6a085731c544a9195caa75 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85f018e-0e70-4987-992f-04d49037d08c_isrForCrosstalkSources_899016_10.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85f018e-0e70-4987-992f-04d49037d08c_isrForCrosstalkSources_899016_10.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 13e822bf068d01208ec051d292cc6216 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9061c5b0-e4b8-44cc-b521-2587c2b56695_isrForCrosstalkSources_899119_4.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9061c5b0-e4b8-44cc-b521-2587c2b56695_isrForCrosstalkSources_899119_4.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 27699ba43e5ae5b9d7e72ed148f084f0 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/947d69a8-0a98-434c-9d9f-98c2a691e295_isrForCrosstalkSources_899016_36.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/947d69a8-0a98-434c-9d9f-98c2a691e295_isrForCrosstalkSources_899016_36.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 52eb985e809be2e6d11b483f62cca57a parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/82bf64d1-b9ba-4341-b573-1dac68600af4_isrForCrosstalkSources_899014_34.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/82bf64d1-b9ba-4341-b573-1dac68600af4_isrForCrosstalkSources_899014_34.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash ecae16763d48e73894ebf2c4f1b56a7e parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437fb83f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d20e4947-6dcd-4abd-9224-5a5c675b5707_isrForCrosstalkSources_899015_15.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d20e4947-6dcd-4abd-9224-5a5c675b5707_isrForCrosstalkSources_899015_15.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 0a34425f18d97a07def6884af7f8c031 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/6b14628a-d656-4f42-95d7-05b8fc014afe_isrForCrosstalkSources_899016_49.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/6b14628a-d656-4f42-95d7-05b8fc014afe_isrForCrosstalkSources_899016_49.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash ff654be4201b09dc731436d25645108a parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/10f76f55-618a-46df-ab9d-5520db8ef62d_isrForCrosstalkSources_899119_30.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/10f76f55-618a-46df-ab9d-5520db8ef62d_isrForCrosstalkSources_899119_30.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 7285c9d34d4efb6b214682d86d90a475 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6327a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f41a71b1-525e-45d3-9e49-5247f137565d_isrForCrosstalkSources_899014_38.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f41a71b1-525e-45d3-9e49-5247f137565d_isrForCrosstalkSources_899014_38.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash cbd4f385bbc1573d1a82f6aadf669910 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6325c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/43d8bd76-993a-4f6a-b9f9-4d9ab8cb29d0_isrForCrosstalkSources_899117_30.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/43d8bd76-993a-4f6a-b9f9-4d9ab8cb29d0_isrForCrosstalkSources_899117_30.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 5ea4016354eda5b5d97528e438b89fe2 parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/27b28b63-a200-4db1-bca3-0a577b83a7a4_isrForCrosstalkSources_899016_32.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/27b28b63-a200-4db1-bca3-0a577b83a7a4_isrForCrosstalkSources_899016_32.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 3ef2c51e7252af8813790c30860da84d parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/32ceec5c-b3a2-4cce-963e-45fb4169a6bf_isrForCrosstalkSources_899015_10.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/32ceec5c-b3a2-4cce-963e-45fb4169a6bf_isrForCrosstalkSources_899015_10.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 90c79cee5eb265c64eaf87d65752e2dd parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f696e1c-63d2-475d-9ccd-b77946dd5245_isrForCrosstalkSources_899015_6.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f696e1c-63d2-475d-9ccd-b77946dd5245_isrForCrosstalkSources_899015_6.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 78b3444bf13272ce676324ce44a833fc parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6323e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f7ffd738-ee89-4398-9a28-a2ee935cb851_isrForCrosstalkSources_899117_49.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f7ffd738-ee89-4398-9a28-a2ee935cb851_isrForCrosstalkSources_899117_49.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash a15bc30cb3447ce04227550058ad8f5b parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/42c2f1a8-910a-48e1-9584-c3067fd0ee41_isrForCrosstalkSources_899119_53.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/42c2f1a8-910a-48e1-9584-c3067fd0ee41_isrForCrosstalkSources_899119_53.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 382d34e64667bb60ac356a30247d7fae parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a7c08419-f84f-4c6b-a063-d9fcb5d9a9dc_isrForCrosstalkSources_899016_42.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a7c08419-f84f-4c6b-a063-d9fcb5d9a9dc_isrForCrosstalkSources_899016_42.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 50f50ac5c6ea5d53f08f8906c8ef8a6c parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4f6aa20f-9280-499a-a051-30688138307f_isrForCrosstalkSources_899117_7.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4f6aa20f-9280-499a-a051-30688138307f_isrForCrosstalkSources_899117_7.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 1817ef000e828bf1f13a98f9a0e5ca0e parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/24fa856e-de9c-432f-a264-b261e308cbe9_isrForCrosstalkSources_899014_21.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/24fa856e-de9c-432f-a264-b261e308cbe9_isrForCrosstalkSources_899014_21.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 2c4a95d43790d2047f4b0a1ed3f87ee0 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d8ed2483-8d60-4f98-8d92-20337dd3e075_isrForCrosstalkSources_899014_48.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d8ed2483-8d60-4f98-8d92-20337dd3e075_isrForCrosstalkSources_899014_48.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 00d6a343c6164930cd2ea8371c5ddbee parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/3fe70fd6-bd69-4b3f-b7ab-8b989bfbbb63_isrForCrosstalkSources_899015_60.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/3fe70fd6-bd69-4b3f-b7ab-8b989bfbbb63_isrForCrosstalkSources_899015_60.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash f8ff5d2661c561e4313718aecd8756d2 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e6218e68-a2a0-4395-8760-3a3a0db5c88d_isrForCrosstalkSources_899015_9.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e6218e68-a2a0-4395-8760-3a3a0db5c88d_isrForCrosstalkSources_899015_9.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash bc7dcfeae3c175a118cc88046c9e6263 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/616f8e07-1d59-406a-a30f-0dd9e78d8209_isrForCrosstalkSources_899014_4.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/616f8e07-1d59-406a-a30f-0dd9e78d8209_isrForCrosstalkSources_899014_4.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 83550e6b43ec910a17df8d3e6b68655f parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a83b7436-fe98-4e28-9d4b-f68206fb0409_isrForCrosstalkSources_899016_8.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a83b7436-fe98-4e28-9d4b-f68206fb0409_isrForCrosstalkSources_899016_8.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash bbedeebaaa287e337a2c835816850438 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e5ef1fa0-7a0f-4b3e-9e54-e5a0ce087b4c_isrForCrosstalkSources_899016_7.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e5ef1fa0-7a0f-4b3e-9e54-e5a0ce087b4c_isrForCrosstalkSources_899016_7.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 3885cf0df557e59ad04eca46d566951b parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f22f8609-65d7-4094-8c52-2ae0d887e89d_isrForCrosstalkSources_899118_49.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f22f8609-65d7-4094-8c52-2ae0d887e89d_isrForCrosstalkSources_899118_49.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 4fea08c0b09f98a5da0356c1177527bd parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d4af1e3-7b3e-49f5-8736-4101085ae5c3_isrForCrosstalkSources_899117_10.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d4af1e3-7b3e-49f5-8736-4101085ae5c3_isrForCrosstalkSources_899117_10.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash c0eced0b35685ca4e6453b557bcd33a0 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb04ccab-f919-46b0-b348-90edba898fba_isrForCrosstalkSources_899016_37.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb04ccab-f919-46b0-b348-90edba898fba_isrForCrosstalkSources_899016_37.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash a502ca931bb0b597d3214332bb8803f0 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/aa2868d7-f52e-4dd4-8e09-460a9d20ceb2_isrForCrosstalkSources_899015_1.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/aa2868d7-f52e-4dd4-8e09-460a9d20ceb2_isrForCrosstalkSources_899015_1.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash b328fcb9f96e137ecb3e89cd1f83e7be parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e5f3d09-9022-4af9-b658-2323451915b2_isrForCrosstalkSources_899014_49.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e5f3d09-9022-4af9-b658-2323451915b2_isrForCrosstalkSources_899014_49.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash f7d643bf5034ad97724c394d26ab09de parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fc14e348-f192-42b4-b6ea-14d74e0a0fae_isrForCrosstalkSources_899118_12.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fc14e348-f192-42b4-b6ea-14d74e0a0fae_isrForCrosstalkSources_899118_12.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash fb5a3306e2d3059d56b5b82ef29f80d7 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/721e5023-9f0b-4d5b-9b94-c5057f0e929b_isrForCrosstalkSources_899014_31.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/721e5023-9f0b-4d5b-9b94-c5057f0e929b_isrForCrosstalkSources_899014_31.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash edef505d3aa3a613bd7538e93691ae47 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e910876d-4aac-4d46-a122-1f27e544bb2b_isrForCrosstalkSources_899119_9.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e910876d-4aac-4d46-a122-1f27e544bb2b_isrForCrosstalkSources_899119_9.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 2539ff32cc5551bf85e82e0e9d734eb1 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6322a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0ca7f5f8-17bb-4bb1-9b84-093003ff580d_isrForCrosstalkSources_899117_38.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0ca7f5f8-17bb-4bb1-9b84-093003ff580d_isrForCrosstalkSources_899117_38.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 53c42ef1c48edd82531a29b1791e19de parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/37df66ec-8bf3-4dc0-8043-806d1a623f94_isrForCrosstalkSources_899016_35.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/37df66ec-8bf3-4dc0-8043-806d1a623f94_isrForCrosstalkSources_899016_35.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash f174c48794bfcb390969c57e5371a61f parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e3c91679-8a5b-40ef-9701-7082c88935f1_isrForCrosstalkSources_899016_40.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e3c91679-8a5b-40ef-9701-7082c88935f1_isrForCrosstalkSources_899016_40.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 26de0ca873e376df846789f6d6aced54 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/85f7b5a8-dcff-49ba-84ef-fcac544a8fba_isrForCrosstalkSources_899119_45.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/85f7b5a8-dcff-49ba-84ef-fcac544a8fba_isrForCrosstalkSources_899119_45.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 3ec3fda3873bf2c5582db86c1e42bbe2 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6300e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5b670ec2-c60c-4bd1-99d1-4a1246c67f1a_isrForCrosstalkSources_899014_50.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5b670ec2-c60c-4bd1-99d1-4a1246c67f1a_isrForCrosstalkSources_899014_50.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 13098bbdd1fb95dbb1a79aa51ce5c0f9 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e37e743-b89a-45b4-a93e-e707466fe813_isrForCrosstalkSources_899016_41.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e37e743-b89a-45b4-a93e-e707466fe813_isrForCrosstalkSources_899016_41.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 6ecee5a112c482b508961342613e3566 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6302c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8c25a158-51f4-4a00-b414-672db6c0b59b_isrForCrosstalkSources_899119_5.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8c25a158-51f4-4a00-b414-672db6c0b59b_isrForCrosstalkSources_899119_5.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash d6fdaafae948532396d51b80ec67584f parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d4935cec-b625-42f3-b2a8-ad0f6c4dc5f3_isrForCrosstalkSources_899015_39.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d4935cec-b625-42f3-b2a8-ad0f6c4dc5f3_isrForCrosstalkSources_899015_39.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash f404c9e5aa3bce02a9052fa13211f5f1 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/65a37345-12c3-4027-b448-55fe3b0ca471_isrForCrosstalkSources_899015_11.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/65a37345-12c3-4027-b448-55fe3b0ca471_isrForCrosstalkSources_899015_11.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 18fd2d2fccd608d7f699ff75b68767ad parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6304a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8d77ae47-43a8-4496-801b-82c26fac38f2_isrForCrosstalkSources_899118_53.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8d77ae47-43a8-4496-801b-82c26fac38f2_isrForCrosstalkSources_899118_53.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash ec8247a145d1d81ba6228baca2198605 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/97e02a0b-5509-4fd8-8cab-dfad9fc3c6e3_isrForCrosstalkSources_899119_44.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/97e02a0b-5509-4fd8-8cab-dfad9fc3c6e3_isrForCrosstalkSources_899119_44.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 421ad2559dfd26db1b9bc3bffaf9f7f8 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/d7049abd-e630-4b88-ad2d-cf8708c6c15a_isrForCrosstalkSources_899016_14.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/d7049abd-e630-4b88-ad2d-cf8708c6c15a_isrForCrosstalkSources_899016_14.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 5c1cdfc4c0026f0043809061bd0f5e6a parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6305e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6cce286e-ba27-4a00-a78e-70103104adb8_isrForCrosstalkSources_899015_2.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6cce286e-ba27-4a00-a78e-70103104adb8_isrForCrosstalkSources_899015_2.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 926e96c5325858b08d77f0a78da7c955 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0c80d14b-d152-459c-8068-150e76c218f7_isrForCrosstalkSources_899119_20.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0c80d14b-d152-459c-8068-150e76c218f7_isrForCrosstalkSources_899119_20.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 459b8e2cf30c66154661d01b3b08f1a1 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6307c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/72460b8d-f5b7-4ac9-85a4-3f9bfe6ab777_isrForCrosstalkSources_899015_31.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/72460b8d-f5b7-4ac9-85a4-3f9bfe6ab777_isrForCrosstalkSources_899015_31.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 079b1b53ba93efade4e9ac7e505f0a88 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/05e0ed46-2316-4e14-82d4-c1eb87227d44_isrForCrosstalkSources_899118_59.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/05e0ed46-2316-4e14-82d4-c1eb87227d44_isrForCrosstalkSources_899118_59.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash c3f10e4123315bcdcb8876a4a3a567ab parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8ef4e7c8-cb11-4b0c-9e37-e17338a89562_isrForCrosstalkSources_899015_24.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8ef4e7c8-cb11-4b0c-9e37-e17338a89562_isrForCrosstalkSources_899015_24.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 8bb5d6b35c630f28d44845a3ee7349f5 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bf308dc1-58d1-4ea0-89a2-4e7589c6c8b8_isrForCrosstalkSources_899014_32.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bf308dc1-58d1-4ea0-89a2-4e7589c6c8b8_isrForCrosstalkSources_899014_32.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 0f1274991ca364166fd89ef3f9846a6d parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6309a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7b6b3a48-95e4-47e5-8787-3a084604b6aa_isrForCrosstalkSources_899119_54.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7b6b3a48-95e4-47e5-8787-3a084604b6aa_isrForCrosstalkSources_899119_54.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 7eff92d607d0b9fd2c5768215c9d894a parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/945ff7ad-162a-42a0-86a6-8f09d56c48cf_isrForCrosstalkSources_899014_52.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/945ff7ad-162a-42a0-86a6-8f09d56c48cf_isrForCrosstalkSources_899014_52.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 7f79756b89417752ac978b09fbaf7c40 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 13:18:57 proc_lsst.multi:170 [INFO] [multi] job_id 17014791 proc_lsst.multi INFO: [multi] job_id 17014791 2024-03-13 13:18:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 13:18:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014791 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014791 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17014791 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-13 13:18:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-13 13:18:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 13:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4742a465-53cd-40fd-b71a-e0ea1b7265fb_isrForCrosstalkSources_899015_27.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4742a465-53cd-40fd-b71a-e0ea1b7265fb_isrForCrosstalkSources_899015_27.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash d1e75e9e02a1c0275cae9105d34199e9 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0b0ea2b2-34f0-4c20-b6d4-e52bf9ff0cf5_isrForCrosstalkSources_899118_28.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0b0ea2b2-34f0-4c20-b6d4-e52bf9ff0cf5_isrForCrosstalkSources_899118_28.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash ca9ab9729d16a8e293a1422eeea43bb7 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5d8fab7d-a840-44c6-bf1d-6de7970357b0_isrForCrosstalkSources_899119_42.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5d8fab7d-a840-44c6-bf1d-6de7970357b0_isrForCrosstalkSources_899119_42.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash d977bcdc8af6fd40c9f19d38a866ad58 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/24f3cd07-ad2e-4537-bbc8-44f330d51c37_isrForCrosstalkSources_899119_18.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/24f3cd07-ad2e-4537-bbc8-44f330d51c37_isrForCrosstalkSources_899119_18.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash bd29ddb8a1073d901a20973ef5c7a7a9 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/eb37a23f-94a2-40a4-b066-91c2702ea8b7_isrForCrosstalkSources_899119_24.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/eb37a23f-94a2-40a4-b066-91c2702ea8b7_isrForCrosstalkSources_899119_24.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 099e36d238e70a15822315dc63871f01 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/2aab18b9-1e8f-475f-b896-f76e725e2312_isrForCrosstalkSources_899118_26.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/2aab18b9-1e8f-475f-b896-f76e725e2312_isrForCrosstalkSources_899118_26.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 547ce61ebab2429889bc3b15b44b6b5a parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e5b2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5c3a2929-fed5-42fe-8ecf-47d0ef1cd616_isrForCrosstalkSources_899117_12.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5c3a2929-fed5-42fe-8ecf-47d0ef1cd616_isrForCrosstalkSources_899117_12.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash b650e9df847df816c202fda9c4244e01 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b41cf6ff-4d34-44de-9899-c97e33dad705_isrForCrosstalkSources_899015_49.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b41cf6ff-4d34-44de-9899-c97e33dad705_isrForCrosstalkSources_899015_49.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 53a1460c1c7a8c4cd29639a71c8c20e8 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/03664809-8059-4d05-b72f-54cfbaffadfd_isrForCrosstalkSources_899118_30.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/03664809-8059-4d05-b72f-54cfbaffadfd_isrForCrosstalkSources_899118_30.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 4a31c98d41affccf27b7da03d38149b6 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/44437d7d-5ed9-41e5-a120-67032631e6ef_isrForCrosstalkSources_899117_59.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/44437d7d-5ed9-41e5-a120-67032631e6ef_isrForCrosstalkSources_899117_59.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 883fbb4d3e6ff7bf381bffbf8987525d parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/64094148-1150-4b9c-af89-964889a70940_isrForCrosstalkSources_899015_55.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/64094148-1150-4b9c-af89-964889a70940_isrForCrosstalkSources_899015_55.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 6b62e740777728506f9fee3663f7126e parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e617c158-4cbe-4d25-a29d-b9eb5a44fce1_isrForCrosstalkSources_899119_48.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e617c158-4cbe-4d25-a29d-b9eb5a44fce1_isrForCrosstalkSources_899119_48.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash e3d9ec793845e307b688d78776dfc331 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/165ca9da-1217-48ab-a61d-89da5a11d8e6_isrForCrosstalkSources_899014_36.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/165ca9da-1217-48ab-a61d-89da5a11d8e6_isrForCrosstalkSources_899014_36.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 0de0efb0359717e989ec783f89a69a72 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/38757443-783e-47f4-9a29-fe39930e069a_isrForCrosstalkSources_899016_13.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/38757443-783e-47f4-9a29-fe39930e069a_isrForCrosstalkSources_899016_13.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 8e51e44536d7100de81ccee5f304153c parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/12ace4ed-990e-459f-8365-c9a5a4b23952_isrForCrosstalkSources_899016_44.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/12ace4ed-990e-459f-8365-c9a5a4b23952_isrForCrosstalkSources_899016_44.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash ae48d69a82a786f61ade65e105982093 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5f93d0df-da75-4371-a4c1-8634d9216aae_isrForCrosstalkSources_899117_6.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5f93d0df-da75-4371-a4c1-8634d9216aae_isrForCrosstalkSources_899117_6.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 22db26109ba9932fd81987666355fb53 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05603725-59e3-44d9-9296-ce4a4a14478c_isrForCrosstalkSources_899117_51.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05603725-59e3-44d9-9296-ce4a4a14478c_isrForCrosstalkSources_899117_51.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash fe15d71d1ef0f145198dfb3722c56562 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0bc9d7b4-f6a6-41a1-8828-dfd728f6c94d_isrForCrosstalkSources_899015_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0bc9d7b4-f6a6-41a1-8828-dfd728f6c94d_isrForCrosstalkSources_899015_19.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 237fdc6835c575808b1b1cdc55f9b8d3 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6320c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a561ccd-0c79-437c-b10f-5306243cc265_isrForCrosstalkSources_899119_46.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a561ccd-0c79-437c-b10f-5306243cc265_isrForCrosstalkSources_899119_46.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 7b1e8d1a7e05a17cc5d8e319bc4a5c7b parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/cc601169-4143-4ee9-8755-8595578c1ee0_isrForCrosstalkSources_899119_3.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/cc601169-4143-4ee9-8755-8595578c1ee0_isrForCrosstalkSources_899119_3.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 2efb85834ecbc674bd746c424bf9cbf9 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d036118f-e7ca-4fb7-aa01-7a520dc6286f_isrForCrosstalkSources_899117_18.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d036118f-e7ca-4fb7-aa01-7a520dc6286f_isrForCrosstalkSources_899117_18.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 60ce023286b7316b8d68757289688e7b parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c85777d6-66a4-436f-b374-e9ad878e62b0_isrForCrosstalkSources_899015_3.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c85777d6-66a4-436f-b374-e9ad878e62b0_isrForCrosstalkSources_899015_3.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 1dd3b23a08ee109b16b982f0274ec89f parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bfb73210-ec99-47a5-9436-8b1241b89115_isrForCrosstalkSources_899014_55.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bfb73210-ec99-47a5-9436-8b1241b89115_isrForCrosstalkSources_899014_55.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash a974f515de6e42655120dca77a7c18e8 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77add4c4-904c-4782-a1ca-47a73640ba40_isrForCrosstalkSources_899117_2.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77add4c4-904c-4782-a1ca-47a73640ba40_isrForCrosstalkSources_899117_2.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 6de9869604918946fb501de7c66875ba parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/986bd71f-1265-4e33-abb1-fcbe754694b2_isrForCrosstalkSources_899117_24.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/986bd71f-1265-4e33-abb1-fcbe754694b2_isrForCrosstalkSources_899117_24.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash af273dcfde85c888019ce40ce60940ce parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/67b2e673-3c03-4861-ab79-c32894210b53_isrForCrosstalkSources_899014_15.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/67b2e673-3c03-4861-ab79-c32894210b53_isrForCrosstalkSources_899014_15.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 49b644ff59c93b73b684b64e67a8d2e3 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f9868757-22c6-422b-9a54-26ccc42064e1_isrForCrosstalkSources_899118_9.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f9868757-22c6-422b-9a54-26ccc42064e1_isrForCrosstalkSources_899118_9.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 4d2345730bdbb1dbca588b3639b5c66f parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/52e005c1-eb0b-43f6-aa0c-fc5b7123b6bc_isrForCrosstalkSources_899016_47.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/52e005c1-eb0b-43f6-aa0c-fc5b7123b6bc_isrForCrosstalkSources_899016_47.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 4184fb4f37f0dfe534629949ab5041bb parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16a11593-1fc9-4b81-b353-1ca859f2da7c_isrForCrosstalkSources_899117_29.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16a11593-1fc9-4b81-b353-1ca859f2da7c_isrForCrosstalkSources_899117_29.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 0e6a34fea8fa9049bcddbe44085f2eb0 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c1cb0b2-fc2c-48a9-ba91-3edc35cda921_isrForCrosstalkSources_899117_61.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c1cb0b2-fc2c-48a9-ba91-3edc35cda921_isrForCrosstalkSources_899117_61.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 91a27b89c1ebd54a52aab37564c99361 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6318a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/50db53df-fbbe-4572-94c4-4507257f12ee_isrForCrosstalkSources_899016_52.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/50db53df-fbbe-4572-94c4-4507257f12ee_isrForCrosstalkSources_899016_52.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash dd3d31a28fe40f9af358a63b9d9a2bf9 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54df4f09-8aa7-467e-92e0-f961b71dda90_isrForCrosstalkSources_899016_21.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54df4f09-8aa7-467e-92e0-f961b71dda90_isrForCrosstalkSources_899016_21.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash b7da6c0acc9c3fafa4bcda3d14f72341 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/dc13ec21-417e-45f5-a18e-a36b4c0af6b7_isrForCrosstalkSources_899117_9.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/dc13ec21-417e-45f5-a18e-a36b4c0af6b7_isrForCrosstalkSources_899117_9.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 8927defb4bcdb53a23fc692c23aa62e5 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6319e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c8af2c72-e35f-4692-9355-b05c5e4fc1da_isrForCrosstalkSources_899016_59.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c8af2c72-e35f-4692-9355-b05c5e4fc1da_isrForCrosstalkSources_899016_59.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash b7d4ed11933473f8c2745f057e70de04 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/33271eb8-869f-4cb6-afbc-8ca80e0938e1_isrForCrosstalkSources_899016_58.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/33271eb8-869f-4cb6-afbc-8ca80e0938e1_isrForCrosstalkSources_899016_58.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash af914b287b1b62d2ba1976296f5ceff9 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a62822c0-f9f8-4031-991e-d5dedb3efcf5_isrForCrosstalkSources_899117_40.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a62822c0-f9f8-4031-991e-d5dedb3efcf5_isrForCrosstalkSources_899117_40.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash a4a5d33b6fb3904099b05297a51ae94a parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6314e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bdae475a-ac33-46ad-a017-dda5378d2113_isrForCrosstalkSources_899118_52.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bdae475a-ac33-46ad-a017-dda5378d2113_isrForCrosstalkSources_899118_52.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 583fcbd0b479fcb3b5573183ddf535d6 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6328e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 13:18:57 proc_lsst.multi:170 [INFO] [multi] job_id 17014792 proc_lsst.multi INFO: [multi] job_id 17014792 2024-03-13 13:18:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 13:18:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014792 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014792 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17014792 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-13 13:18:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-13 13:18:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 13:18:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc1a1d50-ae73-4397-b40f-42156e7839c9_isrForCrosstalkSources_899016_12.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc1a1d50-ae73-4397-b40f-42156e7839c9_isrForCrosstalkSources_899016_12.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash fc85c0ccad1b7fa01f6324c64b2bc6ad parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6316c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/37dda199-d67a-4ef2-8990-a37a7f5f771b_isrForCrosstalkSources_899014_39.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/37dda199-d67a-4ef2-8990-a37a7f5f771b_isrForCrosstalkSources_899014_39.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 9d1c6b77f0a7314704d008bf5e378b58 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6311c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7246c31e-cdff-49c9-aab2-a1c59a593064_isrForCrosstalkSources_899016_31.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7246c31e-cdff-49c9-aab2-a1c59a593064_isrForCrosstalkSources_899016_31.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 61a35390111d2fd2c43a19fde3854e90 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/85f5a619-8196-4f49-b06f-dd2e79f23c60_isrForCrosstalkSources_899014_17.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/85f5a619-8196-4f49-b06f-dd2e79f23c60_isrForCrosstalkSources_899014_17.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash da55fe22d33091e4578aa554f3f203f5 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e630fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4c58acd7-00a1-4520-a8d5-41907e04a5b8_isrForCrosstalkSources_899015_18.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4c58acd7-00a1-4520-a8d5-41907e04a5b8_isrForCrosstalkSources_899015_18.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash a5c21500929c1ec340e0dc7be0400ed1 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e631120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b0590adf-5b88-40f0-b7e0-0e71557bb72f_isrForCrosstalkSources_899015_42.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b0590adf-5b88-40f0-b7e0-0e71557bb72f_isrForCrosstalkSources_899015_42.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash a63439b59b3b1849f329d62a2284726d parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/47b7dba3-1e53-49a7-8f5f-ca45df592195_isrForCrosstalkSources_899016_62.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/47b7dba3-1e53-49a7-8f5f-ca45df592195_isrForCrosstalkSources_899016_62.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 24ae2a03a729204ed5e28e226ee932df parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4c9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/cb8df527-4b8c-44e1-af2f-cb206d7f2fac_isrForCrosstalkSources_899015_47.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/cb8df527-4b8c-44e1-af2f-cb206d7f2fac_isrForCrosstalkSources_899015_47.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 1712192052fd37ec9b23cc0f4bac7228 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/575b81e4-07ce-42fd-84a9-e639aeeb86f3_isrForCrosstalkSources_899014_29.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/575b81e4-07ce-42fd-84a9-e639aeeb86f3_isrForCrosstalkSources_899014_29.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 268aa374878769fae6a6916073ae48d2 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/515ab22a-106c-476c-92fb-001167bb2b7d_isrForCrosstalkSources_899016_2.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/515ab22a-106c-476c-92fb-001167bb2b7d_isrForCrosstalkSources_899016_2.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 33060ae3230edfbda7006e05a6c6f6cd parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6c0556e1-e53c-41da-a52f-2d248c048809_isrForCrosstalkSources_899015_35.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6c0556e1-e53c-41da-a52f-2d248c048809_isrForCrosstalkSources_899015_35.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash d057917a9f9b733a25e2ebf90136b604 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f7cc92f1-45e3-4ea5-b4e9-8d6cf44bb192_isrForCrosstalkSources_899118_40.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f7cc92f1-45e3-4ea5-b4e9-8d6cf44bb192_isrForCrosstalkSources_899118_40.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash c5be3bb26fd822b84730472b621cc677 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5bdb5a77-a00a-4d6f-8e8a-e7bd7a32736c_isrForCrosstalkSources_899015_26.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5bdb5a77-a00a-4d6f-8e8a-e7bd7a32736c_isrForCrosstalkSources_899015_26.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 4b1eeb1730e18422345101497006238b parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b45cc4bd-602c-4070-927b-2376d2da1b5d_isrForCrosstalkSources_899016_20.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b45cc4bd-602c-4070-927b-2376d2da1b5d_isrForCrosstalkSources_899016_20.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash a21cefe62fc8b14d6673935d25cfc5b7 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/caf9c261-e562-46e7-82f1-d992455527a5_isrForCrosstalkSources_899014_37.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/caf9c261-e562-46e7-82f1-d992455527a5_isrForCrosstalkSources_899014_37.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 022dff3bbb0fb6c357e4e526b60d04b4 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1565105a-8237-46a0-ab2f-8bc6c083c20c_isrForCrosstalkSources_899119_61.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1565105a-8237-46a0-ab2f-8bc6c083c20c_isrForCrosstalkSources_899119_61.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 52edf6f83eeca816e13438bcf0418029 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2b12f40e-f93c-4ae7-8ced-759a46663fe0_isrForCrosstalkSources_899014_8.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2b12f40e-f93c-4ae7-8ced-759a46663fe0_isrForCrosstalkSources_899014_8.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 221e304d72ffeb3e55f8ac9bf9a46c44 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e673898e-77a1-4f1a-a1c4-a5117887e639_isrForCrosstalkSources_899117_5.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e673898e-77a1-4f1a-a1c4-a5117887e639_isrForCrosstalkSources_899117_5.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash a08325180f29143f6676a9eee21b1585 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6332e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e3f7ece9-48c0-4f69-857b-8ec7a41c7bd0_isrForCrosstalkSources_899118_51.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e3f7ece9-48c0-4f69-857b-8ec7a41c7bd0_isrForCrosstalkSources_899118_51.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 1a90f91b59c7fa30e21bf2b1d1b6c7a3 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/eda664ab-92da-4125-b023-5838d3de5df1_isrForCrosstalkSources_899118_23.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/eda664ab-92da-4125-b023-5838d3de5df1_isrForCrosstalkSources_899118_23.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash dff3200a45e4b937eae0260c3f1cf0b8 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ad38e85b-0a7a-4a8e-903f-13dc24aac2bc_isrForCrosstalkSources_899016_29.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ad38e85b-0a7a-4a8e-903f-13dc24aac2bc_isrForCrosstalkSources_899016_29.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash d3ecd3cea071639eeb1c437429570558 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6334c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/45cce674-ce12-4fbc-80c3-c6118e5ee168_isrForCrosstalkSources_899117_20.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/45cce674-ce12-4fbc-80c3-c6118e5ee168_isrForCrosstalkSources_899117_20.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash c6d6bff50500e0523b82dbf5250e2011 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6331a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/365d055d-3469-4cac-a582-ff0176ea01f0_isrForCrosstalkSources_899118_22.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/365d055d-3469-4cac-a582-ff0176ea01f0_isrForCrosstalkSources_899118_22.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash a79aa101210756119aa931a01736b941 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1c7881c-96d3-43f9-a3c5-01515ed9af6e_isrForCrosstalkSources_899015_54.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1c7881c-96d3-43f9-a3c5-01515ed9af6e_isrForCrosstalkSources_899015_54.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 31478e8300853890309c1796a9664634 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/284e0c1d-4ebf-4f82-b480-8559c0d3034c_isrForCrosstalkSources_899117_15.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/284e0c1d-4ebf-4f82-b480-8559c0d3034c_isrForCrosstalkSources_899117_15.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash c6c59999ccd95ecc5604029081e0a634 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6336a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f9fa7d23-00de-45e2-b5fa-698d17103818_isrForCrosstalkSources_899014_1.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f9fa7d23-00de-45e2-b5fa-698d17103818_isrForCrosstalkSources_899014_1.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash ded246ff3bbbdcdb40860e84ea2d4e39 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c1b1cecd-4aa2-47ee-962b-e6f58050cfe2_isrForCrosstalkSources_899016_56.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c1b1cecd-4aa2-47ee-962b-e6f58050cfe2_isrForCrosstalkSources_899016_56.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash c581809f6a7acb60f53cae1d9c31cdc5 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/16097181-2f66-44c8-8dbb-1e16ff11e13e_isrForCrosstalkSources_899119_11.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/16097181-2f66-44c8-8dbb-1e16ff11e13e_isrForCrosstalkSources_899119_11.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 62373be29f7857813ea72160152fb2c3 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/83cfa69c-2ae1-4b6f-a149-3f01315653f8_isrForCrosstalkSources_899118_33.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/83cfa69c-2ae1-4b6f-a149-3f01315653f8_isrForCrosstalkSources_899118_33.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash db544bd2766c2c54a7fa08efbf68d380 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-13 13:18:58 proc_lsst.multi:170 [INFO] [multi] job_id 17014793 proc_lsst.multi INFO: [multi] job_id 17014793 2024-03-13 13:18:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:18:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014793 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014793 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17014793 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-13 13:18:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4231e6db-3d04-4678-8e2a-bffdef8e0bc9_isrForCrosstalkSources_899015_61.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4231e6db-3d04-4678-8e2a-bffdef8e0bc9_isrForCrosstalkSources_899015_61.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 1ed83da75e5525dd65de09dc51455705 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/9387cbda-198e-4401-a4f2-c3c4c946976e_isrForCrosstalkSources_899014_44.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/9387cbda-198e-4401-a4f2-c3c4c946976e_isrForCrosstalkSources_899014_44.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 0b6d1a1935bd17741efdbee1749521e8 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6313a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/639e5a63-3dd5-4071-91f1-1f9799045bfd_isrForCrosstalkSources_899117_16.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/639e5a63-3dd5-4071-91f1-1f9799045bfd_isrForCrosstalkSources_899117_16.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 50b7da00440f71bd6510f5b0a4904fa7 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/70f5b3b6-285b-43e0-99e7-8ab21e174082_isrForCrosstalkSources_899119_1.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/70f5b3b6-285b-43e0-99e7-8ab21e174082_isrForCrosstalkSources_899119_1.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash c5fe2114df0004cf913e273b61c58cd1 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bad6cb96-3295-43c1-ab46-daa9616bd16a_isrForCrosstalkSources_899014_58.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bad6cb96-3295-43c1-ab46-daa9616bd16a_isrForCrosstalkSources_899014_58.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 36bceb8bfdbe8cd51887d9c0ef1549b0 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e632c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4e38393f-319f-40d3-b84e-1eb453dc00ff_isrForCrosstalkSources_899015_8.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4e38393f-319f-40d3-b84e-1eb453dc00ff_isrForCrosstalkSources_899015_8.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash c08e7f028f9d4c1f48c4cc93f23983f3 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6339c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e17c9219-099a-4601-ba57-9a0018e98f70_isrForCrosstalkSources_899015_34.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e17c9219-099a-4601-ba57-9a0018e98f70_isrForCrosstalkSources_899015_34.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 9759c70218dbfc8a8b6cf65bcfabf8a8 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8be62aac-8c9d-4a38-805b-68193d625ddd_isrForCrosstalkSources_899016_24.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8be62aac-8c9d-4a38-805b-68193d625ddd_isrForCrosstalkSources_899016_24.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 82b7610fee76ffc0d0fb92ec6d4bffee parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e6337e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/06b4611c-9ecd-4733-b156-8a0787e0ee31_isrForCrosstalkSources_899016_15.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/06b4611c-9ecd-4733-b156-8a0787e0ee31_isrForCrosstalkSources_899016_15.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash addcd5c9ebf99e28e8b7fc9317288dce parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/68594bb6-bfe0-4192-aeef-10f7bc796322_isrForCrosstalkSources_899117_21.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/68594bb6-bfe0-4192-aeef-10f7bc796322_isrForCrosstalkSources_899117_21.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 46f5d031c70a9c20490ff0fbd469a12c parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e633880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/940665f7-f210-4da1-88f0-dc4db22722a9_isrForCrosstalkSources_899118_5.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/940665f7-f210-4da1-88f0-dc4db22722a9_isrForCrosstalkSources_899118_5.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash bf7a0cfe14eaeb67e70938c123dd9952 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4cb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7804a4e3-61c2-4d87-ad00-b3f5150e238c_isrForCrosstalkSources_899016_57.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7804a4e3-61c2-4d87-ad00-b3f5150e238c_isrForCrosstalkSources_899016_57.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 76e37fa7898d55b91d6b0cb33f6ee2fb parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4cb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b9e1d3e6-cdd2-4a2a-9acd-a6deb5d88668_isrForCrosstalkSources_899016_25.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b9e1d3e6-cdd2-4a2a-9acd-a6deb5d88668_isrForCrosstalkSources_899016_25.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash d8c0c688b1c776fbe9b26b8c51ddca70 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4cb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5ae79a11-ba99-48e8-a7cd-68403dad5091_isrForCrosstalkSources_899119_8.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5ae79a11-ba99-48e8-a7cd-68403dad5091_isrForCrosstalkSources_899119_8.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash a08843ba9bd75ac1f3fd22538421d6ac parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4c9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2d110722-ee73-4cc3-a802-51cb94f1c1c9_isrForCrosstalkSources_899015_37.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2d110722-ee73-4cc3-a802-51cb94f1c1c9_isrForCrosstalkSources_899015_37.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash fed249c5f20d92e5fcddcc1bd77c05bb parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4cb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b1c002e-36d0-4543-8ed6-40020cf685b4_isrForCrosstalkSources_899015_23.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b1c002e-36d0-4543-8ed6-40020cf685b4_isrForCrosstalkSources_899015_23.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 365d912d0a4142a4fb39c872d25b9d7f parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4c82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0475067f-8ba7-41e4-bcd4-ad09797020cf_isrForCrosstalkSources_899118_7.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0475067f-8ba7-41e4-bcd4-ad09797020cf_isrForCrosstalkSources_899118_7.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 56016e23555bee6edb81046a8a98f637 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4c80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/45d46e0c-317f-4c35-bda5-58380cc0e63a_isrForCrosstalkSources_899118_50.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/45d46e0c-317f-4c35-bda5-58380cc0e63a_isrForCrosstalkSources_899118_50.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 913d97f4abbf6f295f1b5465df91b6f1 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4c8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54a14e7a-3168-4390-beeb-0aeaea9bf360_isrForCrosstalkSources_899016_17.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54a14e7a-3168-4390-beeb-0aeaea9bf360_isrForCrosstalkSources_899016_17.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 128986b998d254f861914491f8529118 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15437e4c8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e5d716e-223b-4b57-b844-226daca72d37_isrForCrosstalkSources_899016_23.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e5d716e-223b-4b57-b844-226daca72d37_isrForCrosstalkSources_899016_23.stderr 2024-03-13 13:18:58 proc_lsst.multi:170 [INFO] [multi] job_id 17014794 proc_lsst.multi INFO: [multi] job_id 17014794 2024-03-13 13:18:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 13:18:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014794 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014794 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17014794 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-13 13:18:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:18:58 proc_lsst.multi:170 [INFO] [multi] job_id 17014795 proc_lsst.multi INFO: [multi] job_id 17014795 2024-03-13 13:18:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 13:18:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014795 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014795 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17014795 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-13 13:18:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:18:58 proc_lsst.multi:170 [INFO] [multi] job_id 17014796 proc_lsst.multi INFO: [multi] job_id 17014796 2024-03-13 13:18:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 13:18:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014796 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014796 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17014796 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-13 13:18:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 13:18:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:18:59 proc_lsst.multi:170 [INFO] [multi] job_id 17014797 proc_lsst.multi INFO: [multi] job_id 17014797 2024-03-13 13:18:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 13:18:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014797 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014797 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17014797 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-13 13:18:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-13 13:18:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 13:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:18:59 proc_lsst.multi:170 [INFO] [multi] job_id 17014798 proc_lsst.multi INFO: [multi] job_id 17014798 2024-03-13 13:18:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 13:18:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014798 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014798 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17014798 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-13 13:18:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-13 13:18:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:18:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:18:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 13:18:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:18:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:00 proc_lsst.multi:170 [INFO] [multi] job_id 17014799 proc_lsst.multi INFO: [multi] job_id 17014799 2024-03-13 13:19:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 13:19:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014799 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014799 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17014799 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-13 13:19:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-13 13:19:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 13:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:00 proc_lsst.multi:170 [INFO] [multi] job_id 17014800 proc_lsst.multi INFO: [multi] job_id 17014800 2024-03-13 13:19:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 13:19:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014800 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014800 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17014800 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-13 13:19:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-13 13:19:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 13:19:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:01 proc_lsst.multi:170 [INFO] [multi] job_id 17014801 proc_lsst.multi INFO: [multi] job_id 17014801 2024-03-13 13:19:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 13:19:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014801 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014801 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17014801 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-13 13:19:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-13 13:19:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 13:19:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:02 proc_lsst.multi:170 [INFO] [multi] job_id 17014802 proc_lsst.multi INFO: [multi] job_id 17014802 2024-03-13 13:19:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 13:19:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014802 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014802 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17014802 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-13 13:19:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-13 13:19:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 13:19:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:03 proc_lsst.multi:170 [INFO] [multi] job_id 17014803 proc_lsst.multi INFO: [multi] job_id 17014803 2024-03-13 13:19:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 13:19:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014803 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014803 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17014803 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-13 13:19:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-13 13:19:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 13:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:03 proc_lsst.multi:170 [INFO] [multi] job_id 17014804 proc_lsst.multi INFO: [multi] job_id 17014804 2024-03-13 13:19:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 13:19:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014804 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014804 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17014804 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-13 13:19:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-13 13:19:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 13:19:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:04 proc_lsst.multi:170 [INFO] [multi] job_id 17014805 proc_lsst.multi INFO: [multi] job_id 17014805 2024-03-13 13:19:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 13:19:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014805 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014805 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17014805 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-13 13:19:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-13 13:19:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 13:19:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:05 proc_lsst.multi:170 [INFO] [multi] job_id 17014806 proc_lsst.multi INFO: [multi] job_id 17014806 2024-03-13 13:19:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 13:19:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014806 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014806 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17014806 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-13 13:19:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-13 13:19:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 13:19:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:06 proc_lsst.multi:170 [INFO] [multi] job_id 17014807 proc_lsst.multi INFO: [multi] job_id 17014807 2024-03-13 13:19:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 13:19:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014807 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014807 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17014807 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-13 13:19:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-13 13:19:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 13:19:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:07 proc_lsst.multi:170 [INFO] [multi] job_id 17014808 proc_lsst.multi INFO: [multi] job_id 17014808 2024-03-13 13:19:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 13:19:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014808 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014808 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17014808 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-13 13:19:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-13 13:19:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-13 13:19:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5510354f30b87edba4430c5d902e8ee with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed 2024-03-13 13:19:08 proc_lsst.multi:170 [INFO] [multi] job_id 17014809 proc_lsst.multi INFO: [multi] job_id 17014809 2024-03-13 13:19:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 13:19:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014809 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014809 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17014809 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-13 13:19:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-13 13:19:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-13 13:19:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/542f68b2-a3d6-40c6-93c7-7fda584f61ba_isrForCrosstalkSources_899118_27.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/542f68b2-a3d6-40c6-93c7-7fda584f61ba_isrForCrosstalkSources_899118_27.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10ba46bb08967c6afb38d7edc88f7a9f with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d613ce4b-b229-4f96-a048-8063f5439b16_isrForCrosstalkSources_899014_33.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d613ce4b-b229-4f96-a048-8063f5439b16_isrForCrosstalkSources_899014_33.stderr 2024-03-13 13:19:09 proc_lsst.multi:170 [INFO] [multi] job_id 17014810 proc_lsst.multi INFO: [multi] job_id 17014810 2024-03-13 13:19:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 13:19:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014810 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014810 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17014810 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-13 13:19:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-13 13:19:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-13 13:19:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:10 proc_lsst.multi:170 [INFO] [multi] job_id 17014812 proc_lsst.multi INFO: [multi] job_id 17014812 2024-03-13 13:19:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 13:19:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014812 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014812 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17014812 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-13 13:19:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-13 13:19:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-13 13:19:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:11 proc_lsst.multi:170 [INFO] [multi] job_id 17014865 proc_lsst.multi INFO: [multi] job_id 17014865 2024-03-13 13:19:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 13:19:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014865 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014865 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17014865 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-13 13:19:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-13 13:19:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-13 13:19:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3601fd727c651b0d4ab2c95e8716de37 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/877eb286-d6f5-45f4-8f77-04d9f8bfd044_isrForCrosstalkSources_899015_62.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/877eb286-d6f5-45f4-8f77-04d9f8bfd044_isrForCrosstalkSources_899015_62.stderr 2024-03-13 13:19:12 proc_lsst.multi:170 [INFO] [multi] job_id 17014866 proc_lsst.multi INFO: [multi] job_id 17014866 2024-03-13 13:19:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 13:19:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014866 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014866 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17014866 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-13 13:19:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-13 13:19:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-13 13:19:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 424283d585070cd2fefbef4b5f976815 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f637bb7d-9768-498e-a8b5-508367832351_isrForCrosstalkSources_899016_28.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f637bb7d-9768-498e-a8b5-508367832351_isrForCrosstalkSources_899016_28.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b5f4c907285d28af6d26e3a85aae0dd with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0c3bc967-b10c-461a-b5a7-67623945cf32_isrForCrosstalkSources_899118_62.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0c3bc967-b10c-461a-b5a7-67623945cf32_isrForCrosstalkSources_899118_62.stderr 2024-03-13 13:19:13 proc_lsst.multi:170 [INFO] [multi] job_id 17014867 proc_lsst.multi INFO: [multi] job_id 17014867 2024-03-13 13:19:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 13:19:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:13 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014867 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014867 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17014867 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-13 13:19:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-13 13:19:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-13 13:19:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c88f920c0656ef308f7edec1c148979 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/62c435a4-7287-4f60-b848-637dff1d04f2_isrForCrosstalkSources_899014_7.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/62c435a4-7287-4f60-b848-637dff1d04f2_isrForCrosstalkSources_899014_7.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c03dba1ad4ff1f75041a3337b5beb8a2 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a2963aed-25a2-48dc-8a32-a0afc7131109_isrForCrosstalkSources_899117_1.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a2963aed-25a2-48dc-8a32-a0afc7131109_isrForCrosstalkSources_899117_1.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d97206251843ccd69cda06229036db84 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ed8608df-5026-4e7d-9d66-7d16a63c53e7_isrForCrosstalkSources_899117_43.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ed8608df-5026-4e7d-9d66-7d16a63c53e7_isrForCrosstalkSources_899117_43.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f95a8e350ac7be23ae3d6200bcbd6b7d with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9e511fd3-70a0-466a-a0c8-e467fbb7cc95_isrForCrosstalkSources_899119_31.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9e511fd3-70a0-466a-a0c8-e467fbb7cc95_isrForCrosstalkSources_899119_31.stderr 2024-03-13 13:19:15 proc_lsst.multi:170 [INFO] [multi] job_id 17014868 proc_lsst.multi INFO: [multi] job_id 17014868 2024-03-13 13:19:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 13:19:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014868 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014868 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17014868 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-13 13:19:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-13 13:19:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-13 13:19:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4320a304a771da3d1a0310dbce0fd813 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d978f10f-59e5-4b47-a575-66225db035ad_isrForCrosstalkSources_899119_40.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d978f10f-59e5-4b47-a575-66225db035ad_isrForCrosstalkSources_899119_40.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74d3c8d0ce9807ad5dfbe01997db9e71 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/10a38e77-085b-4a26-a1b2-49947617fee1_isrForCrosstalkSources_899014_43.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/10a38e77-085b-4a26-a1b2-49947617fee1_isrForCrosstalkSources_899014_43.stderr 2024-03-13 13:19:16 proc_lsst.multi:170 [INFO] [multi] job_id 17014869 proc_lsst.multi INFO: [multi] job_id 17014869 2024-03-13 13:19:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 13:19:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014869 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014869 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17014869 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-13 13:19:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-13 13:19:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-13 13:19:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:17 proc_lsst.multi:170 [INFO] [multi] job_id 17014870 proc_lsst.multi INFO: [multi] job_id 17014870 2024-03-13 13:19:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 13:19:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014870 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014870 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17014870 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-13 13:19:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-13 13:19:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-13 13:19:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:18 proc_lsst.multi:170 [INFO] [multi] job_id 17014871 proc_lsst.multi INFO: [multi] job_id 17014871 2024-03-13 13:19:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17014871 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17014871 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17014871 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-13 13:19:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-13 13:19:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 13:19:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b489920628722c9c1a143e0941920732 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f6798385-2e83-4853-8e7a-7245ae87954a_isrForCrosstalkSources_899015_59.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f6798385-2e83-4853-8e7a-7245ae87954a_isrForCrosstalkSources_899015_59.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e4d72c406f33eb49a72e1cbafaf4fa0 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/b611d768-f394-4abb-b91b-11d2f95ee218_isrForCrosstalkSources_899117_56.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/b611d768-f394-4abb-b91b-11d2f95ee218_isrForCrosstalkSources_899117_56.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cf8db8af66fa220f0909a015bf85592 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/efb31862-e44d-468c-83a6-a303ee6c738d_isrForCrosstalkSources_899117_36.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/efb31862-e44d-468c-83a6-a303ee6c738d_isrForCrosstalkSources_899117_36.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b58c5f20210cbf9b45973d7708a116ed with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/09d20420-df7b-4c2c-a302-0b1c37dd4ebb_isrForCrosstalkSources_899014_30.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/09d20420-df7b-4c2c-a302-0b1c37dd4ebb_isrForCrosstalkSources_899014_30.stderr 2024-03-13 13:19:20 proc_lsst.multi:170 [INFO] [multi] job_id 17014872 proc_lsst.multi INFO: [multi] job_id 17014872 2024-03-13 13:19:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:20 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17014872 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17014872 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17014872 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-13 13:19:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-13 13:19:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1415b747160d1f85c26b6010c69a7d22 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f5394aeb-c4ef-4123-9338-5fa12bf696ef_isrForCrosstalkSources_899117_60.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f5394aeb-c4ef-4123-9338-5fa12bf696ef_isrForCrosstalkSources_899117_60.stderr 2024-03-13 13:19:21 proc_lsst.multi:170 [INFO] [multi] job_id 17014873 proc_lsst.multi INFO: [multi] job_id 17014873 2024-03-13 13:19:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 13:19:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:21 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17014873 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17014873 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17014873 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-13 13:19:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-13 13:19:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 13:19:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:22 proc_lsst.multi:170 [INFO] [multi] job_id 17014875 proc_lsst.multi INFO: [multi] job_id 17014875 2024-03-13 13:19:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 13:19:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:22 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17014875 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17014875 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17014875 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-13 13:19:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-13 13:19:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 13:19:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55133e5ac2f056027cb4406944dd2567 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f5a9544e-0955-432b-9fc6-6bfefef45e9d_isrForCrosstalkSources_899118_16.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f5a9544e-0955-432b-9fc6-6bfefef45e9d_isrForCrosstalkSources_899118_16.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a3992dc91c6c0662661a1cf53f347e5 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d9af716-134c-4180-b22b-f6df54ffcc2e_isrForCrosstalkSources_899117_44.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d9af716-134c-4180-b22b-f6df54ffcc2e_isrForCrosstalkSources_899117_44.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a576d73bf884cd6f4aa7b11201918c8 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ecef1c84-c53a-4f99-9a7f-7672047e7b38_isrForCrosstalkSources_899118_13.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ecef1c84-c53a-4f99-9a7f-7672047e7b38_isrForCrosstalkSources_899118_13.stderr 2024-03-13 13:19:23 proc_lsst.multi:170 [INFO] [multi] job_id 17014876 proc_lsst.multi INFO: [multi] job_id 17014876 2024-03-13 13:19:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 13:19:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:23 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17014876 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17014876 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17014876 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-13 13:19:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-13 13:19:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 13:19:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry def1015d58dda7a4f0e4da9eb36d9423 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e104b3b0-2d69-4716-98e4-b3b3b079877f_isrForCrosstalkSources_899014_40.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e104b3b0-2d69-4716-98e4-b3b3b079877f_isrForCrosstalkSources_899014_40.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7166137a3dff62fda33ec95d0e0a05ba with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/93ec4364-0aa5-4547-8f0c-2fbfde7f4dcf_isrForCrosstalkSources_899014_25.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/93ec4364-0aa5-4547-8f0c-2fbfde7f4dcf_isrForCrosstalkSources_899014_25.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca636bc57f1200ac2a32359252942f8 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/10097c6f-1206-499f-bf45-1737b480136d_isrForCrosstalkSources_899117_3.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/10097c6f-1206-499f-bf45-1737b480136d_isrForCrosstalkSources_899117_3.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a351fa01bfc5c599397c1f7209a75548 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d2c144f6-6547-453e-8915-fde85522763c_isrForCrosstalkSources_899117_35.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d2c144f6-6547-453e-8915-fde85522763c_isrForCrosstalkSources_899117_35.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b0582266ec6c2af7e1ba1682db1f577 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d22dd43f-7bb8-44c7-8a97-cc2a873e3208_isrForCrosstalkSources_899117_4.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d22dd43f-7bb8-44c7-8a97-cc2a873e3208_isrForCrosstalkSources_899117_4.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 087245b411325b0a6c7aa7c1f6393522 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8fd587ab-3940-41db-a2ee-4c22417e597e_isrForCrosstalkSources_899014_27.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8fd587ab-3940-41db-a2ee-4c22417e597e_isrForCrosstalkSources_899014_27.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d484c044020e83071069564633dd72a with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0cedd62e-0d9f-4308-ae72-a42fc39a683c_isrForCrosstalkSources_899119_33.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0cedd62e-0d9f-4308-ae72-a42fc39a683c_isrForCrosstalkSources_899119_33.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 863dffeb07a8e0e4b1eb6154205af3f5 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c91dff3a-c5a9-4053-b12c-51ea843c5db4_isrForCrosstalkSources_899015_33.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c91dff3a-c5a9-4053-b12c-51ea843c5db4_isrForCrosstalkSources_899015_33.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 674d9f083232f34252f3590e3a562199 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/79fd6eef-afa3-4afa-9417-b2ab12bea07c_isrForCrosstalkSources_899015_53.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/79fd6eef-afa3-4afa-9417-b2ab12bea07c_isrForCrosstalkSources_899015_53.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9efa05f0b167c41036462960bec9cc1e with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28f03f4c-2780-4275-b22c-5eef68520cb4_isrForCrosstalkSources_899014_35.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28f03f4c-2780-4275-b22c-5eef68520cb4_isrForCrosstalkSources_899014_35.stderr 2024-03-13 13:19:25 proc_lsst.multi:170 [INFO] [multi] job_id 17014877 proc_lsst.multi INFO: [multi] job_id 17014877 2024-03-13 13:19:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:19:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:25 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17014877 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17014877 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17014877 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-13 13:19:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-13 13:19:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-13 13:19:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:19:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5f715a9a7295c38cf8cff1f96d0733e with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4a3b0460-7d21-460f-8679-8e55e3d857fa_isrForCrosstalkSources_899117_27.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4a3b0460-7d21-460f-8679-8e55e3d857fa_isrForCrosstalkSources_899117_27.stderr 2024-03-13 13:19:26 proc_lsst.multi:170 [INFO] [multi] job_id 17014878 proc_lsst.multi INFO: [multi] job_id 17014878 2024-03-13 13:19:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:19:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014878 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014878 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17014878 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-13 13:19:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-13 13:19:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:19:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92a957331330c531b0f92733fc23e1e8 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/1a597015-7484-405f-b243-47eda7bbbd89_isrForCrosstalkSources_899014_45.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/1a597015-7484-405f-b243-47eda7bbbd89_isrForCrosstalkSources_899014_45.stderr 2024-03-13 13:19:27 proc_lsst.multi:170 [INFO] [multi] job_id 17014879 proc_lsst.multi INFO: [multi] job_id 17014879 2024-03-13 13:19:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 13:19:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:19:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014879 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014879 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17014879 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-13 13:19:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-13 13:19:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:19:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:19:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:19:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:19:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:19:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:19:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:19:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:19:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-13 13:19:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:19:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-13 13:19:29 proc_lsst.multi:170 [INFO] [multi] job_id 17014880 proc_lsst.multi INFO: [multi] job_id 17014880 2024-03-13 13:19:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 13:19:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:19:29 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014880 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014880 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17014880 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23379652658256 2024-03-13 13:19:29 proc_lsst.multi:146 [INFO] found job 86094 in provider local proc_lsst.multi INFO: found job 86094 in provider local 2024-03-13 13:19:29 proc_lsst.multi:146 [INFO] found job 17014789 in provider astro proc_lsst.multi INFO: found job 17014789 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e29e01f85ec601f3d99ae1abb84371 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0894dfae-4342-4175-8352-df113042dce3_isrForCrosstalkSources_899118_55.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0894dfae-4342-4175-8352-df113042dce3_isrForCrosstalkSources_899118_55.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c8ff39c570ba8eeea50d96107db84da with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/69c8361d-e85a-4303-bded-10122edfc057_isrForCrosstalkSources_899015_13.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/69c8361d-e85a-4303-bded-10122edfc057_isrForCrosstalkSources_899015_13.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c892935a88fd5fc8ac3b13d079f7605 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f59fc51b-010d-4605-a579-70f45c895829_isrForCrosstalkSources_899117_48.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f59fc51b-010d-4605-a579-70f45c895829_isrForCrosstalkSources_899117_48.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5406f371ecbbeab81e0e172af77a9b69 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c0cb8ecc-6e06-4af6-98d3-511a2a4388bb_isrForCrosstalkSources_899119_14.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c0cb8ecc-6e06-4af6-98d3-511a2a4388bb_isrForCrosstalkSources_899119_14.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f406e606f3ccf4e357b0072770357bb5 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ce705733-2c39-4262-bb9a-f305df88b554_isrForCrosstalkSources_899118_24.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ce705733-2c39-4262-bb9a-f305df88b554_isrForCrosstalkSources_899118_24.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a731aea95885fa770a94a22fd82ffe94 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b6d302fb-fb36-48b2-bfed-16e07370d2ce_isrForCrosstalkSources_899119_22.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b6d302fb-fb36-48b2-bfed-16e07370d2ce_isrForCrosstalkSources_899119_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:30 proc_lsst.multi:146 [INFO] found job 17014790 in provider astro proc_lsst.multi INFO: found job 17014790 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e992b389f46d7d686f7774cb04b1af with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8d87e1a1-18e5-4e2d-8047-d5bb831ab705_isrForCrosstalkSources_899015_4.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8d87e1a1-18e5-4e2d-8047-d5bb831ab705_isrForCrosstalkSources_899015_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:31 proc_lsst.multi:146 [INFO] found job 17014791 in provider astro proc_lsst.multi INFO: found job 17014791 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:32 proc_lsst.multi:146 [INFO] found job 17014792 in provider astro proc_lsst.multi INFO: found job 17014792 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcd67ffbb7036e75a8c7b443c0c5ce21 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ffbc126b-1527-40b6-9d86-8d32e25d3ac7_isrForCrosstalkSources_899117_33.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ffbc126b-1527-40b6-9d86-8d32e25d3ac7_isrForCrosstalkSources_899117_33.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48580a4f84922627c2550d99d625aea8 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1d55e837-d9c8-4a56-ba24-dee49151c5ed_isrForCrosstalkSources_899016_34.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1d55e837-d9c8-4a56-ba24-dee49151c5ed_isrForCrosstalkSources_899016_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:34 proc_lsst.multi:146 [INFO] found job 17014793 in provider astro proc_lsst.multi INFO: found job 17014793 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49ec9b7ec32685592b4fa53c21b71905 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/56774c2f-decd-4dc8-b359-f987bba68a88_isrForCrosstalkSources_899117_14.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/56774c2f-decd-4dc8-b359-f987bba68a88_isrForCrosstalkSources_899117_14.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a61472e65aef45c267369e3d09cce8 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/102a342d-7e49-4948-b094-dccb3d122e71_isrForCrosstalkSources_899014_51.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/102a342d-7e49-4948-b094-dccb3d122e71_isrForCrosstalkSources_899014_51.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63038df18c5d0351e47a43206d4ade82 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d05221e4-8f7d-4379-a552-ddba1037e4a8_isrForCrosstalkSources_899118_29.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d05221e4-8f7d-4379-a552-ddba1037e4a8_isrForCrosstalkSources_899118_29.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ac66f4cd85fd845569339ece4187e5a with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5ee9088f-308b-4574-9f68-8846045f7532_isrForCrosstalkSources_899118_3.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5ee9088f-308b-4574-9f68-8846045f7532_isrForCrosstalkSources_899118_3.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73e24406fd71cea6e0df021d112f58c7 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7c9b01b3-bac2-425d-820f-eebaef40f5c2_isrForCrosstalkSources_899117_28.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7c9b01b3-bac2-425d-820f-eebaef40f5c2_isrForCrosstalkSources_899117_28.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c01c84d48b5eb91943c052395d536289 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/dc8eec57-1d7b-42e9-8adb-bdfdfa478137_isrForCrosstalkSources_899016_39.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/dc8eec57-1d7b-42e9-8adb-bdfdfa478137_isrForCrosstalkSources_899016_39.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc1c5811f02e2aab327feef17daf36fb with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/fd08e855-e5d0-4d75-a4f7-9bd9de155cf7_isrForCrosstalkSources_899119_39.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/fd08e855-e5d0-4d75-a4f7-9bd9de155cf7_isrForCrosstalkSources_899119_39.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4455b6f0b8db9fbc8ff34d7c4a7fe101 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a4f6f04-98be-4f40-a529-fe6c05490ec3_isrForCrosstalkSources_899119_12.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a4f6f04-98be-4f40-a529-fe6c05490ec3_isrForCrosstalkSources_899119_12.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43a08380326e1813962b3bc4123cc771 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca206f57-afa7-406c-b730-d248ef51b53e_isrForCrosstalkSources_899016_61.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca206f57-afa7-406c-b730-d248ef51b53e_isrForCrosstalkSources_899016_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:35 proc_lsst.multi:146 [INFO] found job 17014794 in provider astro proc_lsst.multi INFO: found job 17014794 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a9ddbc28ea7af42a6d9ff8d5510c9a4 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/eae04f38-b94b-441a-815c-d27c6cf0d7ba_isrForCrosstalkSources_899015_36.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/eae04f38-b94b-441a-815c-d27c6cf0d7ba_isrForCrosstalkSources_899015_36.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c724f790f639c3420f36559716621a41 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/a2f138aa-4514-4369-b772-c77a4199e545_isrForCrosstalkSources_899014_42.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/a2f138aa-4514-4369-b772-c77a4199e545_isrForCrosstalkSources_899014_42.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6712ace11d8643f8964b6a54e1c3be67 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c98c1e92-2110-43c8-92b4-04e0bf16827b_isrForCrosstalkSources_899119_56.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c98c1e92-2110-43c8-92b4-04e0bf16827b_isrForCrosstalkSources_899119_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:36 proc_lsst.multi:146 [INFO] found job 17014795 in provider astro proc_lsst.multi INFO: found job 17014795 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b146cbdeb593e9a6bd043eb91517e35 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/cffc49ef-2481-4746-a4df-dbf4c453e689_isrForCrosstalkSources_899117_45.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/cffc49ef-2481-4746-a4df-dbf4c453e689_isrForCrosstalkSources_899117_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:37 proc_lsst.multi:146 [INFO] found job 17014796 in provider astro proc_lsst.multi INFO: found job 17014796 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28140ddfa6363d71d6e366e93b1693c4 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8ae49a4f-bd6b-417e-b261-f2a3b7952d17_isrForCrosstalkSources_899118_37.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8ae49a4f-bd6b-417e-b261-f2a3b7952d17_isrForCrosstalkSources_899118_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:39 proc_lsst.multi:146 [INFO] found job 17014797 in provider astro proc_lsst.multi INFO: found job 17014797 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e097f8e2ec10bbc3525ad6b1f2fd181 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/01c2e98a-1706-4407-8d94-d6a25cf000e0_isrForCrosstalkSources_899118_36.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/01c2e98a-1706-4407-8d94-d6a25cf000e0_isrForCrosstalkSources_899118_36.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c021b03b89a3177e313521fc14b00b5 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/3247cf44-4311-46d6-a7b1-b26a6015da64_isrForCrosstalkSources_899119_19.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/3247cf44-4311-46d6-a7b1-b26a6015da64_isrForCrosstalkSources_899119_19.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25b09683baf0a25e18389b3dee1f8989 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b570e4eb-97f6-414a-bb42-720cf75d46ae_isrForCrosstalkSources_899118_14.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b570e4eb-97f6-414a-bb42-720cf75d46ae_isrForCrosstalkSources_899118_14.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6063405466054cf48249a2dde5379a33 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/dd3db005-cabc-471c-92ad-5d94ed5cb009_isrForCrosstalkSources_899014_24.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/dd3db005-cabc-471c-92ad-5d94ed5cb009_isrForCrosstalkSources_899014_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:40 proc_lsst.multi:146 [INFO] found job 17014798 in provider astro proc_lsst.multi INFO: found job 17014798 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb9349c8256d11aff6e3bdb7e62a73d0 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8516ddd-ca74-4354-b4a2-7905dad1965c_isrForCrosstalkSources_899014_23.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8516ddd-ca74-4354-b4a2-7905dad1965c_isrForCrosstalkSources_899014_23.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53b26207fd17479706eb9aca845a2a5 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c1804a3e-99ec-4917-b253-9e7264f1684d_isrForCrosstalkSources_899014_60.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c1804a3e-99ec-4917-b253-9e7264f1684d_isrForCrosstalkSources_899014_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:41 proc_lsst.multi:146 [INFO] found job 17014799 in provider astro proc_lsst.multi INFO: found job 17014799 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:43 proc_lsst.multi:146 [INFO] found job 17014800 in provider astro proc_lsst.multi INFO: found job 17014800 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613995e7bc3a40b825b1c56aaee18aef with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e25a4c19-1c65-453a-b2ef-99a2d244c799_isrForCrosstalkSources_899118_11.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e25a4c19-1c65-453a-b2ef-99a2d244c799_isrForCrosstalkSources_899118_11.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2fe7a2afee91dc668c082e50fdd9dd2 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d28f833a-9e2b-4266-a690-5feff1a4ce0c_isrForCrosstalkSources_899118_60.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d28f833a-9e2b-4266-a690-5feff1a4ce0c_isrForCrosstalkSources_899118_60.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c94c714f739056584c3a0ab03a76b52c with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7addd376-c9d3-4b78-81fa-8d4515557d61_isrForCrosstalkSources_899119_16.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7addd376-c9d3-4b78-81fa-8d4515557d61_isrForCrosstalkSources_899119_16.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf568a32c914551153d9eb49e287392c with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/52f8c8b8-7c72-4683-9d15-9925109e4d51_isrForCrosstalkSources_899119_58.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/52f8c8b8-7c72-4683-9d15-9925109e4d51_isrForCrosstalkSources_899119_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:44 proc_lsst.multi:146 [INFO] found job 17014801 in provider astro proc_lsst.multi INFO: found job 17014801 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8035f45b036a20142b79f59c65babcd with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/34aca2c0-f834-43c5-a7da-0667721eab2b_isrForCrosstalkSources_899014_6.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/34aca2c0-f834-43c5-a7da-0667721eab2b_isrForCrosstalkSources_899014_6.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 679d679ed8848205bca96fca72c58118 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/250711dd-3188-48ce-a5f3-8ac467801301_isrForCrosstalkSources_899015_38.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/250711dd-3188-48ce-a5f3-8ac467801301_isrForCrosstalkSources_899015_38.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecc959a143c915b8cd54ce27d04287ba with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/9076866d-d3dc-425e-8fd4-e788cc19be6a_isrForCrosstalkSources_899016_1.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/9076866d-d3dc-425e-8fd4-e788cc19be6a_isrForCrosstalkSources_899016_1.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93d2e7aa5ee1548adaf141ac80de0c6e with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6fb35236-4754-46e9-a68f-a979eb9a921a_isrForCrosstalkSources_899119_2.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6fb35236-4754-46e9-a68f-a979eb9a921a_isrForCrosstalkSources_899119_2.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78d1f85d2551716f4598cb28505ba8e5 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ef22e199-ab87-45c4-b12e-51d8752dfb24_isrForCrosstalkSources_899014_14.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ef22e199-ab87-45c4-b12e-51d8752dfb24_isrForCrosstalkSources_899014_14.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f45878189c760395e3ee94574015ba with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/1f42ce4c-ada0-465c-b40c-a4666ec206a2_isrForCrosstalkSources_899117_13.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/1f42ce4c-ada0-465c-b40c-a4666ec206a2_isrForCrosstalkSources_899117_13.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16574e4896053176ec4fe646fd25a862 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/293a7ce5-76f7-4ddb-9692-c5444c70e2b9_isrForCrosstalkSources_899016_5.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/293a7ce5-76f7-4ddb-9692-c5444c70e2b9_isrForCrosstalkSources_899016_5.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb02584da0d3fd9f83db282cf5c8fc2f with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6660b3de-6a8a-43ee-b75d-006003dfd9f7_isrForCrosstalkSources_899119_50.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6660b3de-6a8a-43ee-b75d-006003dfd9f7_isrForCrosstalkSources_899119_50.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7074855f4389a464018a44cf8a91ce97 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/42f80eb3-0060-4977-a843-bf55f25f75cf_isrForCrosstalkSources_899117_31.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/42f80eb3-0060-4977-a843-bf55f25f75cf_isrForCrosstalkSources_899117_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:45 proc_lsst.multi:146 [INFO] found job 17014802 in provider astro proc_lsst.multi INFO: found job 17014802 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 378d85020b9ea1c2abf346d4028fcdf1 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f2d7670f-78bc-4c9c-b29e-737f0f38449e_isrForCrosstalkSources_899118_43.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f2d7670f-78bc-4c9c-b29e-737f0f38449e_isrForCrosstalkSources_899118_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:46 proc_lsst.multi:146 [INFO] found job 17014803 in provider astro proc_lsst.multi INFO: found job 17014803 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9167bdfdb2133a14fd058925dd417718 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2df80d40-058b-4e7c-8189-700607790dd3_isrForCrosstalkSources_899014_9.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2df80d40-058b-4e7c-8189-700607790dd3_isrForCrosstalkSources_899014_9.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 266a14ad326d2cb3bac7a29e5b5cb0e7 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/839e4a6f-c602-44e2-ac0f-a26fe1e2528e_isrForCrosstalkSources_899014_11.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/839e4a6f-c602-44e2-ac0f-a26fe1e2528e_isrForCrosstalkSources_899014_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:48 proc_lsst.multi:146 [INFO] found job 17014804 in provider astro proc_lsst.multi INFO: found job 17014804 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:49 proc_lsst.multi:146 [INFO] found job 17014805 in provider astro proc_lsst.multi INFO: found job 17014805 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2a257983bc9a8e69abf132d4afa2880 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a17f534f-4a62-4b50-9228-65a0d4478723_isrForCrosstalkSources_899119_51.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a17f534f-4a62-4b50-9228-65a0d4478723_isrForCrosstalkSources_899119_51.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e5dd2f872e8792f0175d6fbc4ed1841 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4abbef5e-bb29-4069-b5b0-64ef47448e6b_isrForCrosstalkSources_899016_48.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4abbef5e-bb29-4069-b5b0-64ef47448e6b_isrForCrosstalkSources_899016_48.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a39688ef7b7498da95f36f5b9909259 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fd636dc2-9baf-491a-8b63-291e1d6068be_isrForCrosstalkSources_899014_26.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fd636dc2-9baf-491a-8b63-291e1d6068be_isrForCrosstalkSources_899014_26.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfda87b0dc4b65d6c6311925a1d00ebf with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5a607f53-5ee0-4cfb-917b-08caa94e91d2_isrForCrosstalkSources_899118_61.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5a607f53-5ee0-4cfb-917b-08caa94e91d2_isrForCrosstalkSources_899118_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:50 proc_lsst.multi:146 [INFO] found job 17014806 in provider astro proc_lsst.multi INFO: found job 17014806 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:52 proc_lsst.multi:146 [INFO] found job 17014807 in provider astro proc_lsst.multi INFO: found job 17014807 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ceda1fc3fbd37127d99c2cf82f2147d with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2c1cc094-26dc-4bbb-89a8-23dc59dd6c0d_isrForCrosstalkSources_899117_47.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2c1cc094-26dc-4bbb-89a8-23dc59dd6c0d_isrForCrosstalkSources_899117_47.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b437e60e1814591137508a0f208f6c5b with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4a182b7e-d712-42a7-a627-862009730ca1_isrForCrosstalkSources_899015_40.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4a182b7e-d712-42a7-a627-862009730ca1_isrForCrosstalkSources_899015_40.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f4b2d64e5e5e901f502c8c48c1f1bd7 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5b579167-3049-429a-98d2-80ed54bcf50f_isrForCrosstalkSources_899119_6.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5b579167-3049-429a-98d2-80ed54bcf50f_isrForCrosstalkSources_899119_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:53 proc_lsst.multi:146 [INFO] found job 17014808 in provider astro proc_lsst.multi INFO: found job 17014808 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5796b4bc0a0c1e8283d9fb2041849cc with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c0093db4-d24a-4c54-b755-b87dbac83736_isrForCrosstalkSources_899118_17.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c0093db4-d24a-4c54-b755-b87dbac83736_isrForCrosstalkSources_899118_17.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a783dd579ffe06e3ce6b7af82a0f190 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f42a3965-43de-42f9-a7f6-ab6e8baf59f5_isrForCrosstalkSources_899016_46.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f42a3965-43de-42f9-a7f6-ab6e8baf59f5_isrForCrosstalkSources_899016_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:54 proc_lsst.multi:146 [INFO] found job 17014809 in provider astro proc_lsst.multi INFO: found job 17014809 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 663eea81e61c86b3897b30a6a5e0c1f3 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/59acd42b-6d04-4847-8d61-1887d42f97b3_isrForCrosstalkSources_899015_58.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/59acd42b-6d04-4847-8d61-1887d42f97b3_isrForCrosstalkSources_899015_58.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c1584e255a0c258c0998be9149c04cf with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abc10e17-e4f8-4c44-80d0-67c102403fbf_isrForCrosstalkSources_899015_22.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abc10e17-e4f8-4c44-80d0-67c102403fbf_isrForCrosstalkSources_899015_22.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5610bf3a766adada6e328d57a818bca6 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e644f405-2955-4488-9baa-1c9f3ce39d94_isrForCrosstalkSources_899014_53.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e644f405-2955-4488-9baa-1c9f3ce39d94_isrForCrosstalkSources_899014_53.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecd74709033884834e1338dddc5dc9ed with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3835a85f-e485-43ef-bd1a-9895b2ccc28d_isrForCrosstalkSources_899016_4.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3835a85f-e485-43ef-bd1a-9895b2ccc28d_isrForCrosstalkSources_899016_4.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ac7594a650e5397e6c3d83684f5606b with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8318b53a-ad6e-4ece-aaec-8f6534523380_isrForCrosstalkSources_899016_11.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8318b53a-ad6e-4ece-aaec-8f6534523380_isrForCrosstalkSources_899016_11.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9580e0abf29a0056dd9954bfd9bfedf with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/890deb1e-de43-4ecd-bdb0-9afe9c6f3b9b_isrForCrosstalkSources_899117_8.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/890deb1e-de43-4ecd-bdb0-9afe9c6f3b9b_isrForCrosstalkSources_899117_8.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d82fef2cc69cc9375f726874d4d4812c with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8f714d4d-bce0-417b-b79e-6747e3a1662d_isrForCrosstalkSources_899118_25.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8f714d4d-bce0-417b-b79e-6747e3a1662d_isrForCrosstalkSources_899118_25.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 831e2080f4cb36efb120408585cfcf8c with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/969f3a09-de3d-42db-b44d-ec0cf4df1321_isrForCrosstalkSources_899016_54.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/969f3a09-de3d-42db-b44d-ec0cf4df1321_isrForCrosstalkSources_899016_54.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8fa5f3225937a22a6ed67f919ed50ae with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5098cfd4-7c30-446f-849e-c11eab41769d_isrForCrosstalkSources_899117_34.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5098cfd4-7c30-446f-849e-c11eab41769d_isrForCrosstalkSources_899117_34.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70e1336b775cb81867b55ff95f7a765f with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a3a93b53-e915-4709-9e41-50dc8045b897_isrForCrosstalkSources_899015_21.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a3a93b53-e915-4709-9e41-50dc8045b897_isrForCrosstalkSources_899015_21.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2320c3dbd99351f55ef3f01c64506f56 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0e29e968-6cde-41f6-b4c5-ec0e56eca151_isrForCrosstalkSources_899118_2.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0e29e968-6cde-41f6-b4c5-ec0e56eca151_isrForCrosstalkSources_899118_2.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa6160bb9f860fb61d84b44fc8488459 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4c0620c8-1658-42a0-9821-cd473f736c03_isrForCrosstalkSources_899117_46.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4c0620c8-1658-42a0-9821-cd473f736c03_isrForCrosstalkSources_899117_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:55 proc_lsst.multi:146 [INFO] found job 17014810 in provider astro proc_lsst.multi INFO: found job 17014810 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 871bc0f41a72c928b093bd1b0beb6603 with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0551946d-4bf6-4412-bbc9-1cf384bf8986_isrForCrosstalkSources_899119_35.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0551946d-4bf6-4412-bbc9-1cf384bf8986_isrForCrosstalkSources_899119_35.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6014d5dfd039007dc85ebad50e3c5d5 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5cb79f8d-f338-4394-ac39-5b4fec85cd7a_isrForCrosstalkSources_899014_3.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5cb79f8d-f338-4394-ac39-5b4fec85cd7a_isrForCrosstalkSources_899014_3.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2854bdd784c05fda11d4ad42ee4f98f9 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5e94a4bb-8a0a-43ee-965d-18f790b46aa1_isrForCrosstalkSources_899118_8.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5e94a4bb-8a0a-43ee-965d-18f790b46aa1_isrForCrosstalkSources_899118_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:57 proc_lsst.multi:146 [INFO] found job 17014812 in provider astro proc_lsst.multi INFO: found job 17014812 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb39c803541af7b76fdb9add4e8beb9d with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/9121174e-03be-45f2-b48b-04bbdda4094b_isrForCrosstalkSources_899015_52.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/9121174e-03be-45f2-b48b-04bbdda4094b_isrForCrosstalkSources_899015_52.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b73cf7b33ad6e9770d18605866cb5956 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e73f35f-1385-463b-b5ec-3fcaa6769f1e_isrForCrosstalkSources_899014_59.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e73f35f-1385-463b-b5ec-3fcaa6769f1e_isrForCrosstalkSources_899014_59.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec5f58dce02b7be89627584325b6d431 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c8e1038e-041e-4d69-938e-2e8fe505157b_isrForCrosstalkSources_899118_45.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c8e1038e-041e-4d69-938e-2e8fe505157b_isrForCrosstalkSources_899118_45.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9854d4755b4a1a285490a6248c988ed with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/1058b3f3-333a-476a-86b8-975c3ceb9d9f_isrForCrosstalkSources_899118_42.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/1058b3f3-333a-476a-86b8-975c3ceb9d9f_isrForCrosstalkSources_899118_42.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef45da2d69f2ba5c0197278311c8836c with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f20a3648-5a91-4607-8a25-ec02379701ba_isrForCrosstalkSources_899118_6.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f20a3648-5a91-4607-8a25-ec02379701ba_isrForCrosstalkSources_899118_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:58 proc_lsst.multi:146 [INFO] found job 17014865 in provider astro proc_lsst.multi INFO: found job 17014865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:19:59 proc_lsst.multi:146 [INFO] found job 17014866 in provider astro proc_lsst.multi INFO: found job 17014866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45031893ed6eb15f95e275649f38a776 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/bbe8312b-6c42-4674-90ae-26fd66663d7e_isrForCrosstalkSources_899015_32.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/bbe8312b-6c42-4674-90ae-26fd66663d7e_isrForCrosstalkSources_899015_32.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f7e2c7dc4b7a4faabff40343eb90db0 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6234f445-86f7-41de-9a77-d8cf1b61ca45_isrForCrosstalkSources_899119_34.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6234f445-86f7-41de-9a77-d8cf1b61ca45_isrForCrosstalkSources_899119_34.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1184c184dac05c08eeb60f303ae280de with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca753064-9b9b-4441-9876-af88e0c21704_isrForCrosstalkSources_899016_33.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ca753064-9b9b-4441-9876-af88e0c21704_isrForCrosstalkSources_899016_33.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c4f04d4c5d7cf8cceaa37f6582981b0 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc14b9d9-910d-4ee9-aba2-c7f536c689e8_isrForCrosstalkSources_899016_27.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc14b9d9-910d-4ee9-aba2-c7f536c689e8_isrForCrosstalkSources_899016_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:00 proc_lsst.multi:146 [INFO] found job 17014867 in provider astro proc_lsst.multi INFO: found job 17014867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe2da04dbd6a5be049e45908663ae127 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5bc26814-288f-4eae-861a-a5c4a8aa24cd_isrForCrosstalkSources_899014_28.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5bc26814-288f-4eae-861a-a5c4a8aa24cd_isrForCrosstalkSources_899014_28.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20fdf936117b8eae6936a010f9cfb36f with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/23bfcb45-c83e-4521-892d-0657f790903b_isrForCrosstalkSources_899014_12.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/23bfcb45-c83e-4521-892d-0657f790903b_isrForCrosstalkSources_899014_12.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e25e7b40d3e4569a8b96ef6033208d with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/097657a1-6d04-4c89-8873-e04fd2b1b5ea_isrForCrosstalkSources_899117_17.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/097657a1-6d04-4c89-8873-e04fd2b1b5ea_isrForCrosstalkSources_899117_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:01 proc_lsst.multi:146 [INFO] found job 17014868 in provider astro proc_lsst.multi INFO: found job 17014868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f0eb559afde2707543fb5cd95d37fe6 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b40a0a1d-5a16-4d3d-94af-2bbcdcaac047_isrForCrosstalkSources_899014_5.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b40a0a1d-5a16-4d3d-94af-2bbcdcaac047_isrForCrosstalkSources_899014_5.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 969f13bbae10112b7e19a18bde3450e7 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3559d182-aa6a-4d21-8d64-c49de30c112a_isrForCrosstalkSources_899016_55.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/3559d182-aa6a-4d21-8d64-c49de30c112a_isrForCrosstalkSources_899016_55.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e00b53830eba4193280420dfaefa1d8 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d815734c-64a8-4ca7-ad4f-b2a3f458ece5_isrForCrosstalkSources_899119_41.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/d815734c-64a8-4ca7-ad4f-b2a3f458ece5_isrForCrosstalkSources_899119_41.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 382edfa821ebe8a7b90480bd14f6c371 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c70339d0-771c-4c41-bf54-caf08de7b822_isrForCrosstalkSources_899118_54.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/c70339d0-771c-4c41-bf54-caf08de7b822_isrForCrosstalkSources_899118_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:02 proc_lsst.multi:146 [INFO] found job 17014869 in provider astro proc_lsst.multi INFO: found job 17014869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f792f3374bae4b7a2503baf483b75110 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8d282b43-06c9-4dfd-8c82-68c327832a56_isrForCrosstalkSources_899014_62.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/8d282b43-06c9-4dfd-8c82-68c327832a56_isrForCrosstalkSources_899014_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:03 proc_lsst.multi:146 [INFO] found job 17014870 in provider astro proc_lsst.multi INFO: found job 17014870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec846b098fca0ca003165d3b4943769b with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/75db9ec5-898a-4b28-8e07-72b57b25c8fa_isrForCrosstalkSources_899015_5.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/75db9ec5-898a-4b28-8e07-72b57b25c8fa_isrForCrosstalkSources_899015_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:04 proc_lsst.multi:146 [INFO] found job 17014871 in provider astro proc_lsst.multi INFO: found job 17014871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0cb2a957de571ea46147f286ccc3ac with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b2d05f5c-f140-4371-8cce-80a765194ea5_isrForCrosstalkSources_899119_57.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/b2d05f5c-f140-4371-8cce-80a765194ea5_isrForCrosstalkSources_899119_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:05 proc_lsst.multi:146 [INFO] found job 17014872 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30b586d4a35630d42b5acd92a3cdd9cd with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0dcf3bb3-1f21-49d2-b8d5-03d41af03bd2_isrForCrosstalkSources_899118_46.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0dcf3bb3-1f21-49d2-b8d5-03d41af03bd2_isrForCrosstalkSources_899118_46.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a75c8c5f7a6794fbd6715ba70b29410 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/333375cc-4b8e-43b1-a070-dbdddb4154cd_isrForCrosstalkSources_899016_26.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/333375cc-4b8e-43b1-a070-dbdddb4154cd_isrForCrosstalkSources_899016_26.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 009e799693220fc90b46a39dfce7bb1b with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/94bb526a-6759-4085-b54e-695eee2ce630_isrForCrosstalkSources_899015_7.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/94bb526a-6759-4085-b54e-695eee2ce630_isrForCrosstalkSources_899015_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:06 proc_lsst.multi:146 [INFO] found job 17014873 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7865d80bda2e89984641b7d354eba9d with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/dfaacbd7-eeb9-4e38-87aa-b03b5598189e_isrForCrosstalkSources_899119_60.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/dfaacbd7-eeb9-4e38-87aa-b03b5598189e_isrForCrosstalkSources_899119_60.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f7e08f34be2923e370092c37cf1960f with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/45dd7ed4-e6a5-4b98-9b98-a0dc5a8425d0_isrForCrosstalkSources_899015_48.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/45dd7ed4-e6a5-4b98-9b98-a0dc5a8425d0_isrForCrosstalkSources_899015_48.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e01f2df3f7291af4d7ca928c056e7972 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/25cc433f-0c74-4311-96c1-b8b65c8ba1fa_isrForCrosstalkSources_899117_42.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/25cc433f-0c74-4311-96c1-b8b65c8ba1fa_isrForCrosstalkSources_899117_42.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04f52cdcfaf3eb768cf8ca6947bb2604 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e129d99f-fef4-4a55-8d8e-8808ee1e0faa_isrForCrosstalkSources_899014_10.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/e129d99f-fef4-4a55-8d8e-8808ee1e0faa_isrForCrosstalkSources_899014_10.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97a17d5ca7c6a087ad96c99fdc430c83 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/de0fc91e-7696-43b5-9788-09a1f774c4a9_isrForCrosstalkSources_899015_25.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/de0fc91e-7696-43b5-9788-09a1f774c4a9_isrForCrosstalkSources_899015_25.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b3eadb38223804b3059a2039bb78646 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1feabea-42fd-40e6-ad51-0cd4f4e93b38_isrForCrosstalkSources_899015_28.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1feabea-42fd-40e6-ad51-0cd4f4e93b38_isrForCrosstalkSources_899015_28.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3347fef04a2b2089e17f4da6964d64c8 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/73bed8b5-0b94-4199-b4d4-157b0b6ffd9a_isrForCrosstalkSources_899118_10.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/73bed8b5-0b94-4199-b4d4-157b0b6ffd9a_isrForCrosstalkSources_899118_10.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26eaee8f20cfd8470f433fddb893ed6f with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ce955da6-04d2-4492-b5bb-b16d01411dd6_isrForCrosstalkSources_899117_41.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/ce955da6-04d2-4492-b5bb-b16d01411dd6_isrForCrosstalkSources_899117_41.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8b935373252ba4efae3b08274045a35 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/23703ab6-28c7-42fa-b6ed-23e6afe3facc_isrForCrosstalkSources_899119_36.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/23703ab6-28c7-42fa-b6ed-23e6afe3facc_isrForCrosstalkSources_899119_36.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b583f58396bd69fd28b564ad09400651 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/359425fd-5b77-46bf-9532-f77badf8ac8e_isrForCrosstalkSources_899119_52.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/359425fd-5b77-46bf-9532-f77badf8ac8e_isrForCrosstalkSources_899119_52.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 259c1ca2195130e6a0a6e6b427dd96c9 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5dbdee85-47ee-4f13-8623-e9e8e8b1e1eb_isrForCrosstalkSources_899014_2.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5dbdee85-47ee-4f13-8623-e9e8e8b1e1eb_isrForCrosstalkSources_899014_2.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175585b2a95c6c8b77c8fbbb271eed20 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0544773c-6a20-44ca-8e96-c4d850980448_isrForCrosstalkSources_899015_45.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0544773c-6a20-44ca-8e96-c4d850980448_isrForCrosstalkSources_899015_45.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db3c4d6e2bbac2034e16c2750fb75a1 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fbb290aa-9f6c-4a95-837e-d9acaaad2dcd_isrForCrosstalkSources_899014_61.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/fbb290aa-9f6c-4a95-837e-d9acaaad2dcd_isrForCrosstalkSources_899014_61.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59b46f2b901a56afaff58d12446e6dfb with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f50a7519-1692-4bd3-9a62-5ad8323243e2_isrForCrosstalkSources_899014_56.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f50a7519-1692-4bd3-9a62-5ad8323243e2_isrForCrosstalkSources_899014_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:07 proc_lsst.multi:146 [INFO] found job 17014875 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ffd4b205fc4ad6b16c2870f0cf4371e with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ecf977dd-9e22-4e0d-9582-7e384a6da226_isrForCrosstalkSources_899014_22.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ecf977dd-9e22-4e0d-9582-7e384a6da226_isrForCrosstalkSources_899014_22.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df9e45fa8d84edab99e4e986ba559dac with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dc802ae1-a9d5-4ee0-8f1b-d3c57776e38b_isrForCrosstalkSources_899118_15.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dc802ae1-a9d5-4ee0-8f1b-d3c57776e38b_isrForCrosstalkSources_899118_15.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 678d83562059ec94b0c4c09679453a9a with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4f80cb53-de45-4b97-8a00-36121d6475e9_isrForCrosstalkSources_899016_43.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/4f80cb53-de45-4b97-8a00-36121d6475e9_isrForCrosstalkSources_899016_43.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 677bd8179bdb20517c7bd7dc4046eaa3 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/18adbf30-c858-43a0-a632-6126bc1c2ca9_isrForCrosstalkSources_899016_9.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/18adbf30-c858-43a0-a632-6126bc1c2ca9_isrForCrosstalkSources_899016_9.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 487529b9f8db93bb5b431523821ade5a with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/78b01501-74de-43b8-a699-87b56d354bca_isrForCrosstalkSources_899015_17.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/78b01501-74de-43b8-a699-87b56d354bca_isrForCrosstalkSources_899015_17.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70582aa950e60e4bcb03e273ca437ee3 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e6ed2489-93d0-472f-9d7f-bc8580a627f2_isrForCrosstalkSources_899117_23.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e6ed2489-93d0-472f-9d7f-bc8580a627f2_isrForCrosstalkSources_899117_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:09 proc_lsst.multi:146 [INFO] found job 17014876 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014876 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:10 proc_lsst.multi:146 [INFO] found job 17014877 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b5ff0f2d6db720deed941a3e134dde0 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/5851a6da-120c-4c8a-9ec4-ec831d547667_isrForCrosstalkSources_899016_22.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/5851a6da-120c-4c8a-9ec4-ec831d547667_isrForCrosstalkSources_899016_22.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03d9a95346c6de2c70c72c54a4ad968 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f72d04d4-504b-46a1-90d7-a9b83872958c_isrForCrosstalkSources_899015_29.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f72d04d4-504b-46a1-90d7-a9b83872958c_isrForCrosstalkSources_899015_29.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 686b8c84bdd50fe74fb4ead484696147 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b86bd2d9-961c-402c-851d-b3a61e288975_isrForCrosstalkSources_899118_4.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b86bd2d9-961c-402c-851d-b3a61e288975_isrForCrosstalkSources_899118_4.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00314bc404bfb4c1353a49f006d3bb4e with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f6b7ae32-71d4-4616-bdf8-955b9d1991e5_isrForCrosstalkSources_899016_60.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f6b7ae32-71d4-4616-bdf8-955b9d1991e5_isrForCrosstalkSources_899016_60.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 560d8b62552ce5e94ea1aac374126bbd with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5185c45b-0106-428b-8a65-be342f4c1e78_isrForCrosstalkSources_899119_49.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5185c45b-0106-428b-8a65-be342f4c1e78_isrForCrosstalkSources_899119_49.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15742aa5f50b06b6e1bb08ba6c46a651 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b84f07d-c8ff-4f67-a387-5579bbb9af9d_isrForCrosstalkSources_899015_50.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b84f07d-c8ff-4f67-a387-5579bbb9af9d_isrForCrosstalkSources_899015_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:11 proc_lsst.multi:146 [INFO] found job 17014878 in provider escience_a40 proc_lsst.multi INFO: found job 17014878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 697f107138595b64e75e53c478b4035a with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b94b1871-d735-4f24-bd0f-e2d09ec12999_isrForCrosstalkSources_899118_38.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b94b1871-d735-4f24-bd0f-e2d09ec12999_isrForCrosstalkSources_899118_38.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6ed6f303b1b792464f9a37f81f4f144 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5d82899e-e277-402b-8525-e8a657b9218b_isrForCrosstalkSources_899117_26.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5d82899e-e277-402b-8525-e8a657b9218b_isrForCrosstalkSources_899117_26.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 650f60ad3be3a74b20449a198989b5a8 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6f3274e0-6d48-4665-9d9a-30e6639c17dc_isrForCrosstalkSources_899119_43.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/6f3274e0-6d48-4665-9d9a-30e6639c17dc_isrForCrosstalkSources_899119_43.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20d21880b590520294098857c8d41631 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d1079139-28a3-4a78-ac0b-da56d759ff25_isrForCrosstalkSources_899117_62.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d1079139-28a3-4a78-ac0b-da56d759ff25_isrForCrosstalkSources_899117_62.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25a03d613311ccde608ccbefb867ee7 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/776cbde3-8206-4636-8b15-8dfe62a5a942_isrForCrosstalkSources_899119_7.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/776cbde3-8206-4636-8b15-8dfe62a5a942_isrForCrosstalkSources_899119_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:12 proc_lsst.multi:146 [INFO] found job 17014879 in provider escience_a40 proc_lsst.multi INFO: found job 17014879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880' parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68959eea53663cbfa5bdc47f70e5b569 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/47f113c0-ec57-49f1-af68-81444d457379_isrForCrosstalkSources_899119_59.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/47f113c0-ec57-49f1-af68-81444d457379_isrForCrosstalkSources_899119_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:14 proc_lsst.multi:146 [INFO] found job 17014880 in provider escience_a40 proc_lsst.multi INFO: found job 17014880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880' parsl.providers.slurm.slurm DEBUG: squeue returned 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 218 parsl.dataflow.strategy DEBUG: Executor multi has 218 active tasks, 39/0 running/pending blocks, and 40 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 39 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 16 more blocks parsl.executors.status_handling INFO: Scaling out by 16 blocks parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-13 13:20:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-13 13:20:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-13 13:20:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e65414eddd2fafa7bed7171bdeb911 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/505b8fb0-22c9-42ec-945a-9edd258613d2_isrForCrosstalkSources_899015_46.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/505b8fb0-22c9-42ec-945a-9edd258613d2_isrForCrosstalkSources_899015_46.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbb1f3b9bf76e61daf551c9d71e640ab with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b924148b-3e65-4ab3-a15b-f3ebb1d98560_isrForCrosstalkSources_899118_1.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/b924148b-3e65-4ab3-a15b-f3ebb1d98560_isrForCrosstalkSources_899118_1.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11252d614262e570b4663f0b241ec190 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b43c01f9-4d69-42d1-81d9-4937bcc73c78_isrForCrosstalkSources_899016_16.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b43c01f9-4d69-42d1-81d9-4937bcc73c78_isrForCrosstalkSources_899016_16.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da8b19024b034e3233392a0f0998aff7 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a1c80bbb-0399-4093-8ecd-499c245fa66d_isrForCrosstalkSources_899016_50.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a1c80bbb-0399-4093-8ecd-499c245fa66d_isrForCrosstalkSources_899016_50.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d2fa7d3440ab6dfc215f1f33fb1d979 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f5fba590-0967-435b-b1a1-43eb841ae07c_isrForCrosstalkSources_899016_6.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f5fba590-0967-435b-b1a1-43eb841ae07c_isrForCrosstalkSources_899016_6.stderr 2024-03-13 13:20:16 proc_lsst.multi:170 [INFO] [multi] job_id 17014897 proc_lsst.multi INFO: [multi] job_id 17014897 2024-03-13 13:20:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 13:20:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014897 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014897 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17014897 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-13 13:20:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-13 13:20:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-13 13:20:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5e35bbb5f8443e9a7bec719744dcfb7 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/858eff25-409c-4466-8211-0601acf237eb_isrForCrosstalkSources_899119_37.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/858eff25-409c-4466-8211-0601acf237eb_isrForCrosstalkSources_899119_37.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c9c79b9dc29612b34224aca67ac7794 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0760576d-7984-4a2e-8d29-9094b1430222_isrForCrosstalkSources_899119_32.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0760576d-7984-4a2e-8d29-9094b1430222_isrForCrosstalkSources_899119_32.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0183a77f543e8168cfc0a93074a188bf with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ec7ab53f-3c49-4090-8624-b2a664a43e7b_isrForCrosstalkSources_899118_41.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/ec7ab53f-3c49-4090-8624-b2a664a43e7b_isrForCrosstalkSources_899118_41.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fef690f7c03df22faa55ae7b606ed0f with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/331acef6-77b2-4ead-b33c-0cd2873a3615_isrForCrosstalkSources_899117_52.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/331acef6-77b2-4ead-b33c-0cd2873a3615_isrForCrosstalkSources_899117_52.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74828b32565023748f0ce859dd6b24a4 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e0289fc9-3476-4d63-9861-da0ae7a35ffc_isrForCrosstalkSources_899119_10.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e0289fc9-3476-4d63-9861-da0ae7a35ffc_isrForCrosstalkSources_899119_10.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e50dd0a6a33c49dd51c1e6fed3f83b75 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c43a9bc-2c32-43ce-ac5b-0207fb512cb6_isrForCrosstalkSources_899117_22.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c43a9bc-2c32-43ce-ac5b-0207fb512cb6_isrForCrosstalkSources_899117_22.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a75b295006816f47a6f5e86cbc3adcde with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3e83f924-6b1b-4e76-902d-3ab22ff1cf9d_isrForCrosstalkSources_899014_13.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3e83f924-6b1b-4e76-902d-3ab22ff1cf9d_isrForCrosstalkSources_899014_13.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebf3f7b9731b7049b7bad5df73d255bf with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/79472058-007e-4271-9798-dd21dae467c4_isrForCrosstalkSources_899119_29.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/79472058-007e-4271-9798-dd21dae467c4_isrForCrosstalkSources_899119_29.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 405d5e4a5ece4f875449e849c8fc3f31 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/203f1d9d-8498-4775-81ae-85ccbbcfe7aa_isrForCrosstalkSources_899119_47.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/203f1d9d-8498-4775-81ae-85ccbbcfe7aa_isrForCrosstalkSources_899119_47.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bf59d0639b285833931c2f650ffdf41 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/58e00699-000e-4f88-ab0d-0e9c221e1715_isrForCrosstalkSources_899016_38.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/58e00699-000e-4f88-ab0d-0e9c221e1715_isrForCrosstalkSources_899016_38.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e82c72818605a4a43b13f76cf9ca230 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b0b482da-3d4f-484e-928b-00c63e9478b0_isrForCrosstalkSources_899014_47.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/b0b482da-3d4f-484e-928b-00c63e9478b0_isrForCrosstalkSources_899014_47.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a3768ce4467f7b8617c1578e8c499f with result from task 169 2024-03-13 13:20:17 proc_lsst.multi:170 [INFO] [multi] job_id 17014898 proc_lsst.multi INFO: [multi] job_id 17014898 2024-03-13 13:20:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014898 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014898 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17014898 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-13 13:20:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-13 13:20:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f92b8999-4b57-47da-897b-0702d9ff3450_isrForCrosstalkSources_899118_58.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f92b8999-4b57-47da-897b-0702d9ff3450_isrForCrosstalkSources_899118_58.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 694768aa2f154011a6fda44ab542adef with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ab6b987e-1523-40c2-b6b7-16b2eb781d41_isrForCrosstalkSources_899014_16.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/ab6b987e-1523-40c2-b6b7-16b2eb781d41_isrForCrosstalkSources_899014_16.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6cb296b8f0a31b9b6a8902842abd50a with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e4f9ba3b-2c9e-4657-8996-c486f2c39885_isrForCrosstalkSources_899118_20.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e4f9ba3b-2c9e-4657-8996-c486f2c39885_isrForCrosstalkSources_899118_20.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f7f8522560f0b24c5933d37419ea47e with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a9ec1647-204b-448c-9115-18efbe2e26f7_isrForCrosstalkSources_899119_27.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/a9ec1647-204b-448c-9115-18efbe2e26f7_isrForCrosstalkSources_899119_27.stderr 2024-03-13 13:20:19 proc_lsst.multi:170 [INFO] [multi] job_id 17014899 proc_lsst.multi INFO: [multi] job_id 17014899 2024-03-13 13:20:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 13:20:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014899 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014899 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17014899 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-13 13:20:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-13 13:20:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-13 13:20:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bc6191bb25c8917649946b93af5d79b with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/da55fe65-fc5d-4f8d-9469-9a72ad0ccedd_isrForCrosstalkSources_899117_58.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/da55fe65-fc5d-4f8d-9469-9a72ad0ccedd_isrForCrosstalkSources_899117_58.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 056ac9f7c20b6a38fbc1257a053a2888 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8fc9aa2f-2040-4a93-8299-661d96f41d22_isrForCrosstalkSources_899015_51.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8fc9aa2f-2040-4a93-8299-661d96f41d22_isrForCrosstalkSources_899015_51.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09ec83cceb6300faccf94b6b4260ac4f with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3ec04df6-44f7-4ca5-9b75-8c3bdb0f8c9b_isrForCrosstalkSources_899014_46.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/3ec04df6-44f7-4ca5-9b75-8c3bdb0f8c9b_isrForCrosstalkSources_899014_46.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9775cb06dea15c2a5a9a6087731f1e62 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e62edc63-5a77-4d84-bdfb-697fbbcf1d7e_isrForCrosstalkSources_899117_37.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e62edc63-5a77-4d84-bdfb-697fbbcf1d7e_isrForCrosstalkSources_899117_37.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ce796fbb484ffbcf8875783ac02138 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4c171e3b-b35b-4fb9-8352-5e4a82a135a8_isrForCrosstalkSources_899119_28.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4c171e3b-b35b-4fb9-8352-5e4a82a135a8_isrForCrosstalkSources_899119_28.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 086ed435aa3b962adeda8a398b592c0c with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abf779ac-ee35-438c-b1d7-98df252e672b_isrForCrosstalkSources_899015_16.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/abf779ac-ee35-438c-b1d7-98df252e672b_isrForCrosstalkSources_899015_16.stderr 2024-03-13 13:20:20 proc_lsst.multi:170 [INFO] [multi] job_id 17014900 proc_lsst.multi INFO: [multi] job_id 17014900 2024-03-13 13:20:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 13:20:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014900 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014900 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17014900 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-13 13:20:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-13 13:20:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-13 13:20:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6ebbe7d83b22346a911d7a8a9ca33a7 with result from task 182 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/29c782ba-be5e-4bb1-b95e-8866e3da361b_isrForCrosstalkSources_899118_21.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/29c782ba-be5e-4bb1-b95e-8866e3da361b_isrForCrosstalkSources_899118_21.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6d2373ac89def5768c2c2950c7d68a7 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f739003f-ea83-49a3-a2c1-a84233f9266f_isrForCrosstalkSources_899016_18.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f739003f-ea83-49a3-a2c1-a84233f9266f_isrForCrosstalkSources_899016_18.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 209eacf7db3965391d0b8b6150421c9a with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/777df358-7ba0-48ed-a924-4cd0449a242e_isrForCrosstalkSources_899117_54.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/777df358-7ba0-48ed-a924-4cd0449a242e_isrForCrosstalkSources_899117_54.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3e26cb1b34050a07268e7d89964cce4 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/39642c65-d243-4fff-bb3f-8e837dd7d3ba_isrForCrosstalkSources_899015_57.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/39642c65-d243-4fff-bb3f-8e837dd7d3ba_isrForCrosstalkSources_899015_57.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3a2d8a742421c0619abd4bb91b0e371 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/96b07a8d-c208-4c39-853c-f812e90339e6_isrForCrosstalkSources_899119_38.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/96b07a8d-c208-4c39-853c-f812e90339e6_isrForCrosstalkSources_899119_38.stderr 2024-03-13 13:20:21 proc_lsst.multi:170 [INFO] [multi] job_id 17014901 proc_lsst.multi INFO: [multi] job_id 17014901 2024-03-13 13:20:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 13:20:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:21 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014901 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014901 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17014901 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-13 13:20:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-13 13:20:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-13 13:20:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efb3e0aed1bf60a895c80c51d678e64b with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0d808d00-5aa8-4482-89c0-149bbc6487a3_isrForCrosstalkSources_899117_57.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0d808d00-5aa8-4482-89c0-149bbc6487a3_isrForCrosstalkSources_899117_57.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aad21f83a40c24b8518b3be4666a644 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1bac2cc1-1329-4768-b414-55e16a6cf880_isrForCrosstalkSources_899119_17.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1bac2cc1-1329-4768-b414-55e16a6cf880_isrForCrosstalkSources_899119_17.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a333f1683a156e86886973b982b7629a with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c3b782e1-1cb9-4870-9090-9a482070e816_isrForCrosstalkSources_899014_57.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c3b782e1-1cb9-4870-9090-9a482070e816_isrForCrosstalkSources_899014_57.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc9d974fa7ff4f2873827b04ed1aca72 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f1cbbbc-a15a-45e5-ad77-ef47ffeb9480_isrForCrosstalkSources_899015_43.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f1cbbbc-a15a-45e5-ad77-ef47ffeb9480_isrForCrosstalkSources_899015_43.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b780d3e26248307db9a7d76d14a8d0a0 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/dfa67f77-769d-4811-b7aa-bccf715feb25_isrForCrosstalkSources_899015_14.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/dfa67f77-769d-4811-b7aa-bccf715feb25_isrForCrosstalkSources_899015_14.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae249e37f67729676e3f566c33767d58 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8c9f03e-3da6-47d5-a6fb-1ba9fe0e9d1c_isrForCrosstalkSources_899014_41.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f8c9f03e-3da6-47d5-a6fb-1ba9fe0e9d1c_isrForCrosstalkSources_899014_41.stderr 2024-03-13 13:20:23 proc_lsst.multi:170 [INFO] [multi] job_id 17014902 proc_lsst.multi INFO: [multi] job_id 17014902 2024-03-13 13:20:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 13:20:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014902 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014902 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17014902 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-13 13:20:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-13 13:20:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-13 13:20:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b0dfa97e29e8efee2f87e7d3439975e with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/3f10c7e8-e566-48d1-8e41-e6da5a922cda_isrForCrosstalkSources_899118_47.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/3f10c7e8-e566-48d1-8e41-e6da5a922cda_isrForCrosstalkSources_899118_47.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8844d99fcf62a7d21d409e55497992e6 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d85f8ce8-3a5a-4059-bf83-be4af5c25071_isrForCrosstalkSources_899118_44.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d85f8ce8-3a5a-4059-bf83-be4af5c25071_isrForCrosstalkSources_899118_44.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86265aa5e7002144e649ba92dada8982 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f3f836db-3aea-40d4-8daf-ab25ce02145f_isrForCrosstalkSources_899118_32.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f3f836db-3aea-40d4-8daf-ab25ce02145f_isrForCrosstalkSources_899118_32.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e870a1ed5242cfd4e025652aa2f6a11e with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/2dcde062-9304-411a-966b-d56268235327_isrForCrosstalkSources_899119_13.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/2dcde062-9304-411a-966b-d56268235327_isrForCrosstalkSources_899119_13.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a3609371a3db4441f38c836396c075 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2ff13a12-92fb-4bbb-94a8-a81f67520e41_isrForCrosstalkSources_899015_20.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2ff13a12-92fb-4bbb-94a8-a81f67520e41_isrForCrosstalkSources_899015_20.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09b5b10af9040d5f849ece3a4b09c2d6 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/923ffaaa-9c41-434f-95a5-66edb1b7618e_isrForCrosstalkSources_899117_19.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/923ffaaa-9c41-434f-95a5-66edb1b7618e_isrForCrosstalkSources_899117_19.stderr 2024-03-13 13:20:24 proc_lsst.multi:170 [INFO] [multi] job_id 17014903 proc_lsst.multi INFO: [multi] job_id 17014903 2024-03-13 13:20:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 13:20:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014903 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014903 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17014903 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-13 13:20:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-13 13:20:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-13 13:20:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d992e113b82fefce8bd384dcbd3cb830 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c836e4cb-630a-4d32-b190-373221fd7a7d_isrForCrosstalkSources_899119_55.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/c836e4cb-630a-4d32-b190-373221fd7a7d_isrForCrosstalkSources_899119_55.stderr 2024-03-13 13:20:25 proc_lsst.multi:170 [INFO] [multi] job_id 17014904 proc_lsst.multi INFO: [multi] job_id 17014904 2024-03-13 13:20:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 13:20:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014904 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014904 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17014904 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-13 13:20:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-13 13:20:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-13 13:20:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a85a38c020ef84d76863bab75ec74cc with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bf44e037-8286-45a2-bf62-899ef0adfed7_isrForCrosstalkSources_899118_39.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bf44e037-8286-45a2-bf62-899ef0adfed7_isrForCrosstalkSources_899118_39.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43adbf0a2c539ea59f98506ecffea119 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7ee9be3f-e449-434f-b906-ad47a470d171_isrForCrosstalkSources_899015_44.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7ee9be3f-e449-434f-b906-ad47a470d171_isrForCrosstalkSources_899015_44.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14105a262146c79be229f2f658767a19 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/906c27af-81bf-4843-9936-a44d81c0d9e8_isrForCrosstalkSources_899117_25.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/906c27af-81bf-4843-9936-a44d81c0d9e8_isrForCrosstalkSources_899117_25.stderr 2024-03-13 13:20:26 proc_lsst.multi:170 [INFO] [multi] job_id 17014905 proc_lsst.multi INFO: [multi] job_id 17014905 2024-03-13 13:20:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 13:20:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014905 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014905 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17014905 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-13 13:20:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-13 13:20:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-13 13:20:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cde7b6fb5a8222fe7d010881229b2ca with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85a0662-cbeb-495a-a299-05913895491e_isrForCrosstalkSources_899016_3.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85a0662-cbeb-495a-a299-05913895491e_isrForCrosstalkSources_899016_3.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7038173481ea95048faa85f376890df7 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2ca1d32a-1104-4432-ba04-51f660c4ca8d_isrForCrosstalkSources_899117_55.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/2ca1d32a-1104-4432-ba04-51f660c4ca8d_isrForCrosstalkSources_899117_55.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45c7370a5c967ddbeb2280e782a8b90b with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/202a6d61-1bf9-47b0-aa68-e8bd605ceebc_isrForCrosstalkSources_899119_23.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/202a6d61-1bf9-47b0-aa68-e8bd605ceebc_isrForCrosstalkSources_899119_23.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e78ea05ad7dda3675c4ac832a870cf5 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/047b91ee-b310-483a-8380-0c75708b5a91_isrForCrosstalkSources_899119_21.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/047b91ee-b310-483a-8380-0c75708b5a91_isrForCrosstalkSources_899119_21.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a710fa22dab97ad839da34d359aa07a0 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05fd8a2d-b3e8-46e6-83c1-11e88beb0d32_isrForCrosstalkSources_899117_32.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05fd8a2d-b3e8-46e6-83c1-11e88beb0d32_isrForCrosstalkSources_899117_32.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a4bb5a00572489b07ddc2996a6bf4d1 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28418052-5fa8-4f43-af31-4a46100eda3e_isrForCrosstalkSources_899014_19.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/28418052-5fa8-4f43-af31-4a46100eda3e_isrForCrosstalkSources_899014_19.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7898199db8bfe832b611ef0269b22033 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d390164b-3725-4b6b-9212-cc5ac70d50d6_isrForCrosstalkSources_899118_56.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/d390164b-3725-4b6b-9212-cc5ac70d50d6_isrForCrosstalkSources_899118_56.stderr 2024-03-13 13:20:28 proc_lsst.multi:170 [INFO] [multi] job_id 17014906 proc_lsst.multi INFO: [multi] job_id 17014906 2024-03-13 13:20:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 13:20:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:28 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014906 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014906 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17014906 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-13 13:20:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-13 13:20:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-13 13:20:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc75b49b3804c09e90661cde812956c with result from task 209 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/90046cae-8c68-4579-aa10-7b5d238ef888_isrForCrosstalkSources_899016_19.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/90046cae-8c68-4579-aa10-7b5d238ef888_isrForCrosstalkSources_899016_19.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed7713952cca51f62a802f918383db58 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77e9bb1c-c31e-438c-a2f0-2e282290b010_isrForCrosstalkSources_899117_53.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77e9bb1c-c31e-438c-a2f0-2e282290b010_isrForCrosstalkSources_899117_53.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4814f8e9ddd41a2401d561b5d81739 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7c8394b5-8047-485d-947f-290cb5c36953_isrForCrosstalkSources_899016_51.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7c8394b5-8047-485d-947f-290cb5c36953_isrForCrosstalkSources_899016_51.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51b173d1da5c85a2ec485b4ab5bd3ea7 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f305ee7e-ee23-416c-a8e4-352b167fd8dc_isrForCrosstalkSources_899015_41.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/f305ee7e-ee23-416c-a8e4-352b167fd8dc_isrForCrosstalkSources_899015_41.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4de2bc39b9c0e1e5dcb87a4199899b80 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3087576b-ea8e-4f7c-a93a-5940d6ad48c3_isrForCrosstalkSources_899117_50.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3087576b-ea8e-4f7c-a93a-5940d6ad48c3_isrForCrosstalkSources_899117_50.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7980e818c5380e0efa39749098ccee01 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/de7a9fcc-d35e-4051-b62d-9bd023be19ce_isrForCrosstalkSources_899014_20.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/de7a9fcc-d35e-4051-b62d-9bd023be19ce_isrForCrosstalkSources_899014_20.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c69d1977126104f5ae16871faba7525 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/60998f79-7f78-4b77-a94e-63a5ae99c8c0_isrForCrosstalkSources_899118_35.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/60998f79-7f78-4b77-a94e-63a5ae99c8c0_isrForCrosstalkSources_899118_35.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b15e4995307c4a615e5afd23563bfb4e with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/221f32ff-3099-4422-a643-727550686b7d_isrForCrosstalkSources_899118_48.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/221f32ff-3099-4422-a643-727550686b7d_isrForCrosstalkSources_899118_48.stderr 2024-03-13 13:20:29 proc_lsst.multi:170 [INFO] [multi] job_id 17014907 proc_lsst.multi INFO: [multi] job_id 17014907 2024-03-13 13:20:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 13:20:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:29 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014907 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014907 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17014907 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-13 13:20:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-13 13:20:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-13 13:20:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b81a35860cb235dbade8288e21905da with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0e162909-f29e-4026-b8f6-8d08f821b37e_isrForCrosstalkSources_899119_15.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0e162909-f29e-4026-b8f6-8d08f821b37e_isrForCrosstalkSources_899119_15.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1845cbdb68be10024cf9bd1f07420734 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dd558f04-c20d-4cdb-a0a8-dba1a6a45da7_isrForCrosstalkSources_899118_19.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/dd558f04-c20d-4cdb-a0a8-dba1a6a45da7_isrForCrosstalkSources_899118_19.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d48821c13b9ba2f7085cfc12a60cf13 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c1c027de-bd77-41b2-b2bc-1cbdd8cfaef8_isrForCrosstalkSources_899015_12.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c1c027de-bd77-41b2-b2bc-1cbdd8cfaef8_isrForCrosstalkSources_899015_12.stderr 2024-03-13 13:20:30 proc_lsst.multi:170 [INFO] [multi] job_id 17014908 proc_lsst.multi INFO: [multi] job_id 17014908 2024-03-13 13:20:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 13:20:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:30 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014908 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014908 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17014908 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-13 13:20:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-13 13:20:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-13 13:20:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d146ee24ea5a0efe4654a2491e1a9d66 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/973bd1c8-03cf-4dab-896b-dc7e714dbf43_isrForCrosstalkSources_899119_62.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/973bd1c8-03cf-4dab-896b-dc7e714dbf43_isrForCrosstalkSources_899119_62.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b3561fa12f98cd7205043ac3ac733ba with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fe5da567-c815-4bb6-af2e-9dbe1cd7bed2_isrForCrosstalkSources_899118_57.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fe5da567-c815-4bb6-af2e-9dbe1cd7bed2_isrForCrosstalkSources_899118_57.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee4ceb80c413cfb176ede5de24ea6f86 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/1e30a0a0-5aea-4532-b78b-6554b021eb58_isrForCrosstalkSources_899015_30.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/1e30a0a0-5aea-4532-b78b-6554b021eb58_isrForCrosstalkSources_899015_30.stderr 2024-03-13 13:20:32 proc_lsst.multi:170 [INFO] [multi] job_id 17014909 proc_lsst.multi INFO: [multi] job_id 17014909 2024-03-13 13:20:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 13:20:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:32 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014909 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014909 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17014909 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-13 13:20:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-13 13:20:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-13 13:20:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1304e0def7391e87ed9c0a1e0c36a3b4 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5cbe0414-6ccd-41d0-a090-4043bfbc0647_isrForCrosstalkSources_899119_26.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5cbe0414-6ccd-41d0-a090-4043bfbc0647_isrForCrosstalkSources_899119_26.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27699ba43e5ae5b9d7e72ed148f084f0 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/947d69a8-0a98-434c-9d9f-98c2a691e295_isrForCrosstalkSources_899016_36.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/947d69a8-0a98-434c-9d9f-98c2a691e295_isrForCrosstalkSources_899016_36.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4aac257a9189f23ae3e1fee893a2565b with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fd743f40-fa09-4310-b600-2480100f9406_isrForCrosstalkSources_899016_45.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fd743f40-fa09-4310-b600-2480100f9406_isrForCrosstalkSources_899016_45.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58edbdd1913767b4f86df38b2f91a172 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb63602d-909f-4ca2-b0c2-c6f5531d8799_isrForCrosstalkSources_899016_30.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb63602d-909f-4ca2-b0c2-c6f5531d8799_isrForCrosstalkSources_899016_30.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52eb985e809be2e6d11b483f62cca57a with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/82bf64d1-b9ba-4341-b573-1dac68600af4_isrForCrosstalkSources_899014_34.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/82bf64d1-b9ba-4341-b573-1dac68600af4_isrForCrosstalkSources_899014_34.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e822bf068d01208ec051d292cc6216 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9061c5b0-e4b8-44cc-b521-2587c2b56695_isrForCrosstalkSources_899119_4.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/9061c5b0-e4b8-44cc-b521-2587c2b56695_isrForCrosstalkSources_899119_4.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecae16763d48e73894ebf2c4f1b56a7e with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d20e4947-6dcd-4abd-9224-5a5c675b5707_isrForCrosstalkSources_899015_15.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d20e4947-6dcd-4abd-9224-5a5c675b5707_isrForCrosstalkSources_899015_15.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 216e7cc8d69287a45ec0f5ac3234a0b3 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8d3a879d-f8aa-4061-8e6a-14930e5149cc_isrForCrosstalkSources_899119_25.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8d3a879d-f8aa-4061-8e6a-14930e5149cc_isrForCrosstalkSources_899119_25.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5e626904fa96155e2851e9489ec5b00 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5e57e2b3-f845-4601-b012-8ebc427eecd6_isrForCrosstalkSources_899015_56.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5e57e2b3-f845-4601-b012-8ebc427eecd6_isrForCrosstalkSources_899015_56.stderr 2024-03-13 13:20:33 proc_lsst.multi:170 [INFO] [multi] job_id 17014910 proc_lsst.multi INFO: [multi] job_id 17014910 2024-03-13 13:20:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 13:20:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:33 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014910 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014910 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17014910 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-13 13:20:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-13 13:20:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-13 13:20:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1f68bb26bae4b80fc4da5da273a5d84 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16cf4b91-083e-4e72-bfba-b881f2a20156_isrForCrosstalkSources_899117_39.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16cf4b91-083e-4e72-bfba-b881f2a20156_isrForCrosstalkSources_899117_39.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cec316ae9e7e143c1f60eae029185f0 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5453c8f4-fabd-40a3-a81d-55a132ac57c8_isrForCrosstalkSources_899118_34.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/5453c8f4-fabd-40a3-a81d-55a132ac57c8_isrForCrosstalkSources_899118_34.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71f91129fa5d05f11ba703c102dd5061 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f494afb4-f0da-4912-9e1f-eef1fdc2fc25_isrForCrosstalkSources_899014_18.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f494afb4-f0da-4912-9e1f-eef1fdc2fc25_isrForCrosstalkSources_899014_18.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f6a1f4f2a6a085731c544a9195caa75 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85f018e-0e70-4987-992f-04d49037d08c_isrForCrosstalkSources_899016_10.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/f85f018e-0e70-4987-992f-04d49037d08c_isrForCrosstalkSources_899016_10.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9239611b4c53d08f3cd4402aa78a4755 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/166d73c3-fb5a-477a-a350-4a9c0d2c88fc_isrForCrosstalkSources_899016_53.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/166d73c3-fb5a-477a-a350-4a9c0d2c88fc_isrForCrosstalkSources_899016_53.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d10429b7324c7f4662749b81c9d45f85 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7db8b61c-187c-4f53-9046-90f16349b1ea_isrForCrosstalkSources_899117_11.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/7db8b61c-187c-4f53-9046-90f16349b1ea_isrForCrosstalkSources_899117_11.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b30c50606f6ea0fff619ecaaad46851 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/566c1e18-52a3-418a-ad42-fd6136b62845_isrForCrosstalkSources_899118_18.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/566c1e18-52a3-418a-ad42-fd6136b62845_isrForCrosstalkSources_899118_18.stderr 2024-03-13 13:20:34 proc_lsst.multi:170 [INFO] [multi] job_id 17014911 proc_lsst.multi INFO: [multi] job_id 17014911 2024-03-13 13:20:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 13:20:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014911 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014911 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17014911 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-13 13:20:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-13 13:20:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-13 13:20:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-13 13:20:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-13 13:20:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-13 13:20:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-13 13:20:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-13 13:20:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-13 13:20:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-13 13:20:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-13 13:20:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,n3009,10.64.65.9,127.0.0.1,169.254.95.120 -p 0 -c 1.0 -m None --poll 10 --task_port=54259 --result_port=54939 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f7be5a81e62c583a965b11991f6b233 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/7634d1d8-e451-4876-8333-15a7eee51922_isrForCrosstalkSources_899118_31.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/7634d1d8-e451-4876-8333-15a7eee51922_isrForCrosstalkSources_899118_31.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7285c9d34d4efb6b214682d86d90a475 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f41a71b1-525e-45d3-9e49-5247f137565d_isrForCrosstalkSources_899014_38.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f41a71b1-525e-45d3-9e49-5247f137565d_isrForCrosstalkSources_899014_38.stderr 2024-03-13 13:20:36 proc_lsst.multi:170 [INFO] [multi] job_id 17014912 proc_lsst.multi INFO: [multi] job_id 17014912 2024-03-13 13:20:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-13 13:20:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-13 13:20:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17014912 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17014912 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17014912 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23379652658256 2024-03-13 13:20:36 proc_lsst.multi:146 [INFO] found job 86094 in provider local proc_lsst.multi INFO: found job 86094 in provider local parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a34425f18d97a07def6884af7f8c031 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/6b14628a-d656-4f42-95d7-05b8fc014afe_isrForCrosstalkSources_899016_49.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/6b14628a-d656-4f42-95d7-05b8fc014afe_isrForCrosstalkSources_899016_49.stderr 2024-03-13 13:20:36 proc_lsst.multi:146 [INFO] found job 17014789 in provider astro proc_lsst.multi INFO: found job 17014789 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe743bb299565eb9269091e3a05eadde with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c6d3b75b-0033-4e13-b666-63f32136dbd4_isrForCrosstalkSources_899014_54.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/c6d3b75b-0033-4e13-b666-63f32136dbd4_isrForCrosstalkSources_899014_54.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff654be4201b09dc731436d25645108a with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/10f76f55-618a-46df-ab9d-5520db8ef62d_isrForCrosstalkSources_899119_30.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/10f76f55-618a-46df-ab9d-5520db8ef62d_isrForCrosstalkSources_899119_30.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ef2c51e7252af8813790c30860da84d with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/32ceec5c-b3a2-4cce-963e-45fb4169a6bf_isrForCrosstalkSources_899015_10.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/32ceec5c-b3a2-4cce-963e-45fb4169a6bf_isrForCrosstalkSources_899015_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:37 proc_lsst.multi:146 [INFO] found job 17014790 in provider astro proc_lsst.multi INFO: found job 17014790 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90c79cee5eb265c64eaf87d65752e2dd with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f696e1c-63d2-475d-9ccd-b77946dd5245_isrForCrosstalkSources_899015_6.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/7f696e1c-63d2-475d-9ccd-b77946dd5245_isrForCrosstalkSources_899015_6.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fea08c0b09f98a5da0356c1177527bd with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d4af1e3-7b3e-49f5-8736-4101085ae5c3_isrForCrosstalkSources_899117_10.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/3d4af1e3-7b3e-49f5-8736-4101085ae5c3_isrForCrosstalkSources_899117_10.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbd4f385bbc1573d1a82f6aadf669910 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/43d8bd76-993a-4f6a-b9f9-4d9ab8cb29d0_isrForCrosstalkSources_899117_30.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/43d8bd76-993a-4f6a-b9f9-4d9ab8cb29d0_isrForCrosstalkSources_899117_30.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78b3444bf13272ce676324ce44a833fc with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f7ffd738-ee89-4398-9a28-a2ee935cb851_isrForCrosstalkSources_899117_49.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/f7ffd738-ee89-4398-9a28-a2ee935cb851_isrForCrosstalkSources_899117_49.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ea4016354eda5b5d97528e438b89fe2 with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/27b28b63-a200-4db1-bca3-0a577b83a7a4_isrForCrosstalkSources_899016_32.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/27b28b63-a200-4db1-bca3-0a577b83a7a4_isrForCrosstalkSources_899016_32.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0eced0b35685ca4e6453b557bcd33a0 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb04ccab-f919-46b0-b348-90edba898fba_isrForCrosstalkSources_899016_37.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/fb04ccab-f919-46b0-b348-90edba898fba_isrForCrosstalkSources_899016_37.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a502ca931bb0b597d3214332bb8803f0 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/aa2868d7-f52e-4dd4-8e09-460a9d20ceb2_isrForCrosstalkSources_899015_1.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/aa2868d7-f52e-4dd4-8e09-460a9d20ceb2_isrForCrosstalkSources_899015_1.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50f50ac5c6ea5d53f08f8906c8ef8a6c with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4f6aa20f-9280-499a-a051-30688138307f_isrForCrosstalkSources_899117_7.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/4f6aa20f-9280-499a-a051-30688138307f_isrForCrosstalkSources_899117_7.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 382d34e64667bb60ac356a30247d7fae with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a7c08419-f84f-4c6b-a063-d9fcb5d9a9dc_isrForCrosstalkSources_899016_42.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a7c08419-f84f-4c6b-a063-d9fcb5d9a9dc_isrForCrosstalkSources_899016_42.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b328fcb9f96e137ecb3e89cd1f83e7be with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e5f3d09-9022-4af9-b658-2323451915b2_isrForCrosstalkSources_899014_49.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2e5f3d09-9022-4af9-b658-2323451915b2_isrForCrosstalkSources_899014_49.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1817ef000e828bf1f13a98f9a0e5ca0e with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/24fa856e-de9c-432f-a264-b261e308cbe9_isrForCrosstalkSources_899014_21.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/24fa856e-de9c-432f-a264-b261e308cbe9_isrForCrosstalkSources_899014_21.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a15bc30cb3447ce04227550058ad8f5b with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/42c2f1a8-910a-48e1-9584-c3067fd0ee41_isrForCrosstalkSources_899119_53.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/42c2f1a8-910a-48e1-9584-c3067fd0ee41_isrForCrosstalkSources_899119_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:38 proc_lsst.multi:146 [INFO] found job 17014791 in provider astro proc_lsst.multi INFO: found job 17014791 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c4a95d43790d2047f4b0a1ed3f87ee0 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d8ed2483-8d60-4f98-8d92-20337dd3e075_isrForCrosstalkSources_899014_48.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/d8ed2483-8d60-4f98-8d92-20337dd3e075_isrForCrosstalkSources_899014_48.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc7dcfeae3c175a118cc88046c9e6263 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/616f8e07-1d59-406a-a30f-0dd9e78d8209_isrForCrosstalkSources_899014_4.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/616f8e07-1d59-406a-a30f-0dd9e78d8209_isrForCrosstalkSources_899014_4.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00d6a343c6164930cd2ea8371c5ddbee with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/3fe70fd6-bd69-4b3f-b7ab-8b989bfbbb63_isrForCrosstalkSources_899015_60.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/3fe70fd6-bd69-4b3f-b7ab-8b989bfbbb63_isrForCrosstalkSources_899015_60.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83550e6b43ec910a17df8d3e6b68655f with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a83b7436-fe98-4e28-9d4b-f68206fb0409_isrForCrosstalkSources_899016_8.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/a83b7436-fe98-4e28-9d4b-f68206fb0409_isrForCrosstalkSources_899016_8.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ff5d2661c561e4313718aecd8756d2 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e6218e68-a2a0-4395-8760-3a3a0db5c88d_isrForCrosstalkSources_899015_9.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e6218e68-a2a0-4395-8760-3a3a0db5c88d_isrForCrosstalkSources_899015_9.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7d643bf5034ad97724c394d26ab09de with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fc14e348-f192-42b4-b6ea-14d74e0a0fae_isrForCrosstalkSources_899118_12.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/fc14e348-f192-42b4-b6ea-14d74e0a0fae_isrForCrosstalkSources_899118_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:39 proc_lsst.multi:146 [INFO] found job 17014792 in provider astro proc_lsst.multi INFO: found job 17014792 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edef505d3aa3a613bd7538e93691ae47 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e910876d-4aac-4d46-a122-1f27e544bb2b_isrForCrosstalkSources_899119_9.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e910876d-4aac-4d46-a122-1f27e544bb2b_isrForCrosstalkSources_899119_9.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb5a3306e2d3059d56b5b82ef29f80d7 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/721e5023-9f0b-4d5b-9b94-c5057f0e929b_isrForCrosstalkSources_899014_31.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/721e5023-9f0b-4d5b-9b94-c5057f0e929b_isrForCrosstalkSources_899014_31.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6fdaafae948532396d51b80ec67584f with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d4935cec-b625-42f3-b2a8-ad0f6c4dc5f3_isrForCrosstalkSources_899015_39.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/d4935cec-b625-42f3-b2a8-ad0f6c4dc5f3_isrForCrosstalkSources_899015_39.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2539ff32cc5551bf85e82e0e9d734eb1 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0ca7f5f8-17bb-4bb1-9b84-093003ff580d_isrForCrosstalkSources_899117_38.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/0ca7f5f8-17bb-4bb1-9b84-093003ff580d_isrForCrosstalkSources_899117_38.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ec3fda3873bf2c5582db86c1e42bbe2 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5b670ec2-c60c-4bd1-99d1-4a1246c67f1a_isrForCrosstalkSources_899014_50.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/5b670ec2-c60c-4bd1-99d1-4a1246c67f1a_isrForCrosstalkSources_899014_50.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f174c48794bfcb390969c57e5371a61f with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e3c91679-8a5b-40ef-9701-7082c88935f1_isrForCrosstalkSources_899016_40.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e3c91679-8a5b-40ef-9701-7082c88935f1_isrForCrosstalkSources_899016_40.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13098bbdd1fb95dbb1a79aa51ce5c0f9 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e37e743-b89a-45b4-a93e-e707466fe813_isrForCrosstalkSources_899016_41.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e37e743-b89a-45b4-a93e-e707466fe813_isrForCrosstalkSources_899016_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:41 proc_lsst.multi:146 [INFO] found job 17014793 in provider astro proc_lsst.multi INFO: found job 17014793 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 421ad2559dfd26db1b9bc3bffaf9f7f8 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/d7049abd-e630-4b88-ad2d-cf8708c6c15a_isrForCrosstalkSources_899016_14.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/d7049abd-e630-4b88-ad2d-cf8708c6c15a_isrForCrosstalkSources_899016_14.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec8247a145d1d81ba6228baca2198605 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/97e02a0b-5509-4fd8-8cab-dfad9fc3c6e3_isrForCrosstalkSources_899119_44.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/97e02a0b-5509-4fd8-8cab-dfad9fc3c6e3_isrForCrosstalkSources_899119_44.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18fd2d2fccd608d7f699ff75b68767ad with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8d77ae47-43a8-4496-801b-82c26fac38f2_isrForCrosstalkSources_899118_53.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/8d77ae47-43a8-4496-801b-82c26fac38f2_isrForCrosstalkSources_899118_53.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 079b1b53ba93efade4e9ac7e505f0a88 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/05e0ed46-2316-4e14-82d4-c1eb87227d44_isrForCrosstalkSources_899118_59.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/05e0ed46-2316-4e14-82d4-c1eb87227d44_isrForCrosstalkSources_899118_59.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f404c9e5aa3bce02a9052fa13211f5f1 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/65a37345-12c3-4027-b448-55fe3b0ca471_isrForCrosstalkSources_899015_11.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/65a37345-12c3-4027-b448-55fe3b0ca471_isrForCrosstalkSources_899015_11.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53c42ef1c48edd82531a29b1791e19de with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/37df66ec-8bf3-4dc0-8043-806d1a623f94_isrForCrosstalkSources_899016_35.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/37df66ec-8bf3-4dc0-8043-806d1a623f94_isrForCrosstalkSources_899016_35.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3f10e4123315bcdcb8876a4a3a567ab with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8ef4e7c8-cb11-4b0c-9e37-e17338a89562_isrForCrosstalkSources_899015_24.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8ef4e7c8-cb11-4b0c-9e37-e17338a89562_isrForCrosstalkSources_899015_24.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26de0ca873e376df846789f6d6aced54 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/85f7b5a8-dcff-49ba-84ef-fcac544a8fba_isrForCrosstalkSources_899119_45.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/85f7b5a8-dcff-49ba-84ef-fcac544a8fba_isrForCrosstalkSources_899119_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:42 proc_lsst.multi:146 [INFO] found job 17014794 in provider astro proc_lsst.multi INFO: found job 17014794 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ecee5a112c482b508961342613e3566 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8c25a158-51f4-4a00-b414-672db6c0b59b_isrForCrosstalkSources_899119_5.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/8c25a158-51f4-4a00-b414-672db6c0b59b_isrForCrosstalkSources_899119_5.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bb5d6b35c630f28d44845a3ee7349f5 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bf308dc1-58d1-4ea0-89a2-4e7589c6c8b8_isrForCrosstalkSources_899014_32.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bf308dc1-58d1-4ea0-89a2-4e7589c6c8b8_isrForCrosstalkSources_899014_32.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 926e96c5325858b08d77f0a78da7c955 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0c80d14b-d152-459c-8068-150e76c218f7_isrForCrosstalkSources_899119_20.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/0c80d14b-d152-459c-8068-150e76c218f7_isrForCrosstalkSources_899119_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:43 proc_lsst.multi:146 [INFO] found job 17014795 in provider astro proc_lsst.multi INFO: found job 17014795 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f79756b89417752ac978b09fbaf7c40 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4742a465-53cd-40fd-b71a-e0ea1b7265fb_isrForCrosstalkSources_899015_27.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4742a465-53cd-40fd-b71a-e0ea1b7265fb_isrForCrosstalkSources_899015_27.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 459b8e2cf30c66154661d01b3b08f1a1 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/72460b8d-f5b7-4ac9-85a4-3f9bfe6ab777_isrForCrosstalkSources_899015_31.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/72460b8d-f5b7-4ac9-85a4-3f9bfe6ab777_isrForCrosstalkSources_899015_31.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d977bcdc8af6fd40c9f19d38a866ad58 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/24f3cd07-ad2e-4537-bbc8-44f330d51c37_isrForCrosstalkSources_899119_18.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/24f3cd07-ad2e-4537-bbc8-44f330d51c37_isrForCrosstalkSources_899119_18.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd29ddb8a1073d901a20973ef5c7a7a9 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/eb37a23f-94a2-40a4-b066-91c2702ea8b7_isrForCrosstalkSources_899119_24.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/eb37a23f-94a2-40a4-b066-91c2702ea8b7_isrForCrosstalkSources_899119_24.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca9ab9729d16a8e293a1422eeea43bb7 with result from task 288 parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:44 proc_lsst.multi:146 [INFO] found job 17014796 in provider astro proc_lsst.multi INFO: found job 17014796 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5d8fab7d-a840-44c6-bf1d-6de7970357b0_isrForCrosstalkSources_899119_42.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5d8fab7d-a840-44c6-bf1d-6de7970357b0_isrForCrosstalkSources_899119_42.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3885cf0df557e59ad04eca46d566951b with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f22f8609-65d7-4094-8c52-2ae0d887e89d_isrForCrosstalkSources_899118_49.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f22f8609-65d7-4094-8c52-2ae0d887e89d_isrForCrosstalkSources_899118_49.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1e75e9e02a1c0275cae9105d34199e9 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0b0ea2b2-34f0-4c20-b6d4-e52bf9ff0cf5_isrForCrosstalkSources_899118_28.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0b0ea2b2-34f0-4c20-b6d4-e52bf9ff0cf5_isrForCrosstalkSources_899118_28.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1274991ca364166fd89ef3f9846a6d with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7b6b3a48-95e4-47e5-8787-3a084604b6aa_isrForCrosstalkSources_899119_54.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/7b6b3a48-95e4-47e5-8787-3a084604b6aa_isrForCrosstalkSources_899119_54.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbedeebaaa287e337a2c835816850438 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e5ef1fa0-7a0f-4b3e-9e54-e5a0ce087b4c_isrForCrosstalkSources_899016_7.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/e5ef1fa0-7a0f-4b3e-9e54-e5a0ce087b4c_isrForCrosstalkSources_899016_7.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c1cdfc4c0026f0043809061bd0f5e6a with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6cce286e-ba27-4a00-a78e-70103104adb8_isrForCrosstalkSources_899015_2.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6cce286e-ba27-4a00-a78e-70103104adb8_isrForCrosstalkSources_899015_2.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eff92d607d0b9fd2c5768215c9d894a with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/945ff7ad-162a-42a0-86a6-8f09d56c48cf_isrForCrosstalkSources_899014_52.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/945ff7ad-162a-42a0-86a6-8f09d56c48cf_isrForCrosstalkSources_899014_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:46 proc_lsst.multi:146 [INFO] found job 17014797 in provider astro proc_lsst.multi INFO: found job 17014797 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 099e36d238e70a15822315dc63871f01 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/2aab18b9-1e8f-475f-b896-f76e725e2312_isrForCrosstalkSources_899118_26.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/2aab18b9-1e8f-475f-b896-f76e725e2312_isrForCrosstalkSources_899118_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:47 proc_lsst.multi:146 [INFO] found job 17014798 in provider astro proc_lsst.multi INFO: found job 17014798 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 547ce61ebab2429889bc3b15b44b6b5a with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5c3a2929-fed5-42fe-8ecf-47d0ef1cd616_isrForCrosstalkSources_899117_12.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5c3a2929-fed5-42fe-8ecf-47d0ef1cd616_isrForCrosstalkSources_899117_12.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a31c98d41affccf27b7da03d38149b6 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/44437d7d-5ed9-41e5-a120-67032631e6ef_isrForCrosstalkSources_899117_59.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/44437d7d-5ed9-41e5-a120-67032631e6ef_isrForCrosstalkSources_899117_59.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 883fbb4d3e6ff7bf381bffbf8987525d with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/64094148-1150-4b9c-af89-964889a70940_isrForCrosstalkSources_899015_55.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/64094148-1150-4b9c-af89-964889a70940_isrForCrosstalkSources_899015_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:48 proc_lsst.multi:146 [INFO] found job 17014799 in provider astro proc_lsst.multi INFO: found job 17014799 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22db26109ba9932fd81987666355fb53 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05603725-59e3-44d9-9296-ce4a4a14478c_isrForCrosstalkSources_899117_51.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/05603725-59e3-44d9-9296-ce4a4a14478c_isrForCrosstalkSources_899117_51.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e51e44536d7100de81ccee5f304153c with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/12ace4ed-990e-459f-8365-c9a5a4b23952_isrForCrosstalkSources_899016_44.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/12ace4ed-990e-459f-8365-c9a5a4b23952_isrForCrosstalkSources_899016_44.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe15d71d1ef0f145198dfb3722c56562 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0bc9d7b4-f6a6-41a1-8828-dfd728f6c94d_isrForCrosstalkSources_899015_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/0bc9d7b4-f6a6-41a1-8828-dfd728f6c94d_isrForCrosstalkSources_899015_19.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0de0efb0359717e989ec783f89a69a72 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/38757443-783e-47f4-9a29-fe39930e069a_isrForCrosstalkSources_899016_13.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/38757443-783e-47f4-9a29-fe39930e069a_isrForCrosstalkSources_899016_13.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4a5d33b6fb3904099b05297a51ae94a with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bdae475a-ac33-46ad-a017-dda5378d2113_isrForCrosstalkSources_899118_52.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/bdae475a-ac33-46ad-a017-dda5378d2113_isrForCrosstalkSources_899118_52.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d4ed11933473f8c2745f057e70de04 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/33271eb8-869f-4cb6-afbc-8ca80e0938e1_isrForCrosstalkSources_899016_58.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/33271eb8-869f-4cb6-afbc-8ca80e0938e1_isrForCrosstalkSources_899016_58.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b1e8d1a7e05a17cc5d8e319bc4a5c7b with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/cc601169-4143-4ee9-8755-8595578c1ee0_isrForCrosstalkSources_899119_3.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/cc601169-4143-4ee9-8755-8595578c1ee0_isrForCrosstalkSources_899119_3.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ce023286b7316b8d68757289688e7b with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c85777d6-66a4-436f-b374-e9ad878e62b0_isrForCrosstalkSources_899015_3.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/c85777d6-66a4-436f-b374-e9ad878e62b0_isrForCrosstalkSources_899015_3.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2efb85834ecbc674bd746c424bf9cbf9 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d036118f-e7ca-4fb7-aa01-7a520dc6286f_isrForCrosstalkSources_899117_18.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/d036118f-e7ca-4fb7-aa01-7a520dc6286f_isrForCrosstalkSources_899117_18.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da55fe22d33091e4578aa554f3f203f5 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4c58acd7-00a1-4520-a8d5-41907e04a5b8_isrForCrosstalkSources_899015_18.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4c58acd7-00a1-4520-a8d5-41907e04a5b8_isrForCrosstalkSources_899015_18.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a1460c1c7a8c4cd29639a71c8c20e8 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/03664809-8059-4d05-b72f-54cfbaffadfd_isrForCrosstalkSources_899118_30.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/03664809-8059-4d05-b72f-54cfbaffadfd_isrForCrosstalkSources_899118_30.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dd3b23a08ee109b16b982f0274ec89f with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bfb73210-ec99-47a5-9436-8b1241b89115_isrForCrosstalkSources_899014_55.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bfb73210-ec99-47a5-9436-8b1241b89115_isrForCrosstalkSources_899014_55.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 583fcbd0b479fcb3b5573183ddf535d6 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc1a1d50-ae73-4397-b40f-42156e7839c9_isrForCrosstalkSources_899016_12.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/bc1a1d50-ae73-4397-b40f-42156e7839c9_isrForCrosstalkSources_899016_12.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4184fb4f37f0dfe534629949ab5041bb with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16a11593-1fc9-4b81-b353-1ca859f2da7c_isrForCrosstalkSources_899117_29.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/16a11593-1fc9-4b81-b353-1ca859f2da7c_isrForCrosstalkSources_899117_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:49 proc_lsst.multi:146 [INFO] found job 17014800 in provider astro proc_lsst.multi INFO: found job 17014800 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e6a34fea8fa9049bcddbe44085f2eb0 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c1cb0b2-fc2c-48a9-ba91-3edc35cda921_isrForCrosstalkSources_899117_61.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/9c1cb0b2-fc2c-48a9-ba91-3edc35cda921_isrForCrosstalkSources_899117_61.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61a35390111d2fd2c43a19fde3854e90 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/85f5a619-8196-4f49-b06f-dd2e79f23c60_isrForCrosstalkSources_899014_17.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/85f5a619-8196-4f49-b06f-dd2e79f23c60_isrForCrosstalkSources_899014_17.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc85c0ccad1b7fa01f6324c64b2bc6ad with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/37dda199-d67a-4ef2-8990-a37a7f5f771b_isrForCrosstalkSources_899014_39.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/37dda199-d67a-4ef2-8990-a37a7f5f771b_isrForCrosstalkSources_899014_39.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d1c6b77f0a7314704d008bf5e378b58 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7246c31e-cdff-49c9-aab2-a1c59a593064_isrForCrosstalkSources_899016_31.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7246c31e-cdff-49c9-aab2-a1c59a593064_isrForCrosstalkSources_899016_31.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3d9ec793845e307b688d78776dfc331 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/165ca9da-1217-48ab-a61d-89da5a11d8e6_isrForCrosstalkSources_899014_36.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/165ca9da-1217-48ab-a61d-89da5a11d8e6_isrForCrosstalkSources_899014_36.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af914b287b1b62d2ba1976296f5ceff9 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a62822c0-f9f8-4031-991e-d5dedb3efcf5_isrForCrosstalkSources_899117_40.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/a62822c0-f9f8-4031-991e-d5dedb3efcf5_isrForCrosstalkSources_899117_40.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8927defb4bcdb53a23fc692c23aa62e5 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c8af2c72-e35f-4692-9355-b05c5e4fc1da_isrForCrosstalkSources_899016_59.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c8af2c72-e35f-4692-9355-b05c5e4fc1da_isrForCrosstalkSources_899016_59.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7da6c0acc9c3fafa4bcda3d14f72341 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/dc13ec21-417e-45f5-a18e-a36b4c0af6b7_isrForCrosstalkSources_899117_9.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/dc13ec21-417e-45f5-a18e-a36b4c0af6b7_isrForCrosstalkSources_899117_9.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1712192052fd37ec9b23cc0f4bac7228 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/575b81e4-07ce-42fd-84a9-e639aeeb86f3_isrForCrosstalkSources_899014_29.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/575b81e4-07ce-42fd-84a9-e639aeeb86f3_isrForCrosstalkSources_899014_29.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5c21500929c1ec340e0dc7be0400ed1 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b0590adf-5b88-40f0-b7e0-0e71557bb72f_isrForCrosstalkSources_899015_42.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b0590adf-5b88-40f0-b7e0-0e71557bb72f_isrForCrosstalkSources_899015_42.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a63439b59b3b1849f329d62a2284726d with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/47b7dba3-1e53-49a7-8f5f-ca45df592195_isrForCrosstalkSources_899016_62.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/47b7dba3-1e53-49a7-8f5f-ca45df592195_isrForCrosstalkSources_899016_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:51 proc_lsst.multi:146 [INFO] found job 17014801 in provider astro proc_lsst.multi INFO: found job 17014801 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae48d69a82a786f61ade65e105982093 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5f93d0df-da75-4371-a4c1-8634d9216aae_isrForCrosstalkSources_899117_6.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/5f93d0df-da75-4371-a4c1-8634d9216aae_isrForCrosstalkSources_899117_6.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b650e9df847df816c202fda9c4244e01 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b41cf6ff-4d34-44de-9899-c97e33dad705_isrForCrosstalkSources_899015_49.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/b41cf6ff-4d34-44de-9899-c97e33dad705_isrForCrosstalkSources_899015_49.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b62e740777728506f9fee3663f7126e with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e617c158-4cbe-4d25-a29d-b9eb5a44fce1_isrForCrosstalkSources_899119_48.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/e617c158-4cbe-4d25-a29d-b9eb5a44fce1_isrForCrosstalkSources_899119_48.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24ae2a03a729204ed5e28e226ee932df with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/cb8df527-4b8c-44e1-af2f-cb206d7f2fac_isrForCrosstalkSources_899015_47.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/cb8df527-4b8c-44e1-af2f-cb206d7f2fac_isrForCrosstalkSources_899015_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:52 proc_lsst.multi:146 [INFO] found job 17014802 in provider astro proc_lsst.multi INFO: found job 17014802 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 268aa374878769fae6a6916073ae48d2 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/515ab22a-106c-476c-92fb-001167bb2b7d_isrForCrosstalkSources_899016_2.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/515ab22a-106c-476c-92fb-001167bb2b7d_isrForCrosstalkSources_899016_2.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33060ae3230edfbda7006e05a6c6f6cd with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6c0556e1-e53c-41da-a52f-2d248c048809_isrForCrosstalkSources_899015_35.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/6c0556e1-e53c-41da-a52f-2d248c048809_isrForCrosstalkSources_899015_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:53 proc_lsst.multi:146 [INFO] found job 17014803 in provider astro proc_lsst.multi INFO: found job 17014803 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5be3bb26fd822b84730472b621cc677 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5bdb5a77-a00a-4d6f-8e8a-e7bd7a32736c_isrForCrosstalkSources_899015_26.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/5bdb5a77-a00a-4d6f-8e8a-e7bd7a32736c_isrForCrosstalkSources_899015_26.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a21cefe62fc8b14d6673935d25cfc5b7 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/caf9c261-e562-46e7-82f1-d992455527a5_isrForCrosstalkSources_899014_37.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/caf9c261-e562-46e7-82f1-d992455527a5_isrForCrosstalkSources_899014_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:54 proc_lsst.multi:146 [INFO] found job 17014804 in provider astro proc_lsst.multi INFO: found job 17014804 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:56 proc_lsst.multi:146 [INFO] found job 17014805 in provider astro proc_lsst.multi INFO: found job 17014805 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50b7da00440f71bd6510f5b0a4904fa7 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/70f5b3b6-285b-43e0-99e7-8ab21e174082_isrForCrosstalkSources_899119_1.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/70f5b3b6-285b-43e0-99e7-8ab21e174082_isrForCrosstalkSources_899119_1.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82b7610fee76ffc0d0fb92ec6d4bffee with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/06b4611c-9ecd-4733-b156-8a0787e0ee31_isrForCrosstalkSources_899016_15.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/06b4611c-9ecd-4733-b156-8a0787e0ee31_isrForCrosstalkSources_899016_15.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31478e8300853890309c1796a9664634 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/284e0c1d-4ebf-4f82-b480-8559c0d3034c_isrForCrosstalkSources_899117_15.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/284e0c1d-4ebf-4f82-b480-8559c0d3034c_isrForCrosstalkSources_899117_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:57 proc_lsst.multi:146 [INFO] found job 17014806 in provider astro proc_lsst.multi INFO: found job 17014806 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5fe2114df0004cf913e273b61c58cd1 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bad6cb96-3295-43c1-ab46-daa9616bd16a_isrForCrosstalkSources_899014_58.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/bad6cb96-3295-43c1-ab46-daa9616bd16a_isrForCrosstalkSources_899014_58.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6c59999ccd95ecc5604029081e0a634 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f9fa7d23-00de-45e2-b5fa-698d17103818_isrForCrosstalkSources_899014_1.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/f9fa7d23-00de-45e2-b5fa-698d17103818_isrForCrosstalkSources_899014_1.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a08843ba9bd75ac1f3fd22538421d6ac with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2d110722-ee73-4cc3-a802-51cb94f1c1c9_isrForCrosstalkSources_899015_37.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/2d110722-ee73-4cc3-a802-51cb94f1c1c9_isrForCrosstalkSources_899015_37.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ded246ff3bbbdcdb40860e84ea2d4e39 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c1b1cecd-4aa2-47ee-962b-e6f58050cfe2_isrForCrosstalkSources_899016_56.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/c1b1cecd-4aa2-47ee-962b-e6f58050cfe2_isrForCrosstalkSources_899016_56.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 365d912d0a4142a4fb39c872d25b9d7f with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0475067f-8ba7-41e4-bcd4-ad09797020cf_isrForCrosstalkSources_899118_7.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/0475067f-8ba7-41e4-bcd4-ad09797020cf_isrForCrosstalkSources_899118_7.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fed249c5f20d92e5fcddcc1bd77c05bb with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b1c002e-36d0-4543-8ed6-40020cf685b4_isrForCrosstalkSources_899015_23.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/8b1c002e-36d0-4543-8ed6-40020cf685b4_isrForCrosstalkSources_899015_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:58 proc_lsst.multi:146 [INFO] found job 17014807 in provider astro proc_lsst.multi INFO: found job 17014807 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62373be29f7857813ea72160152fb2c3 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/83cfa69c-2ae1-4b6f-a149-3f01315653f8_isrForCrosstalkSources_899118_33.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/83cfa69c-2ae1-4b6f-a149-3f01315653f8_isrForCrosstalkSources_899118_33.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b6d1a1935bd17741efdbee1749521e8 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/639e5a63-3dd5-4071-91f1-1f9799045bfd_isrForCrosstalkSources_899117_16.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/639e5a63-3dd5-4071-91f1-1f9799045bfd_isrForCrosstalkSources_899117_16.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed83da75e5525dd65de09dc51455705 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/9387cbda-198e-4401-a4f2-c3c4c946976e_isrForCrosstalkSources_899014_44.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/9387cbda-198e-4401-a4f2-c3c4c946976e_isrForCrosstalkSources_899014_44.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08e7f028f9d4c1f48c4cc93f23983f3 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e17c9219-099a-4601-ba57-9a0018e98f70_isrForCrosstalkSources_899015_34.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/e17c9219-099a-4601-ba57-9a0018e98f70_isrForCrosstalkSources_899015_34.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9759c70218dbfc8a8b6cf65bcfabf8a8 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8be62aac-8c9d-4a38-805b-68193d625ddd_isrForCrosstalkSources_899016_24.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/8be62aac-8c9d-4a38-805b-68193d625ddd_isrForCrosstalkSources_899016_24.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8c0c688b1c776fbe9b26b8c51ddca70 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5ae79a11-ba99-48e8-a7cd-68403dad5091_isrForCrosstalkSources_899119_8.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/5ae79a11-ba99-48e8-a7cd-68403dad5091_isrForCrosstalkSources_899119_8.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf7a0cfe14eaeb67e70938c123dd9952 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7804a4e3-61c2-4d87-ad00-b3f5150e238c_isrForCrosstalkSources_899016_57.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/7804a4e3-61c2-4d87-ad00-b3f5150e238c_isrForCrosstalkSources_899016_57.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36bceb8bfdbe8cd51887d9c0ef1549b0 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4e38393f-319f-40d3-b84e-1eb453dc00ff_isrForCrosstalkSources_899015_8.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4e38393f-319f-40d3-b84e-1eb453dc00ff_isrForCrosstalkSources_899015_8.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db544bd2766c2c54a7fa08efbf68d380 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4231e6db-3d04-4678-8e2a-bffdef8e0bc9_isrForCrosstalkSources_899015_61.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/4231e6db-3d04-4678-8e2a-bffdef8e0bc9_isrForCrosstalkSources_899015_61.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c581809f6a7acb60f53cae1d9c31cdc5 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/16097181-2f66-44c8-8dbb-1e16ff11e13e_isrForCrosstalkSources_899119_11.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/16097181-2f66-44c8-8dbb-1e16ff11e13e_isrForCrosstalkSources_899119_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:59 proc_lsst.multi:146 [INFO] found job 17014808 in provider astro proc_lsst.multi INFO: found job 17014808 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46f5d031c70a9c20490ff0fbd469a12c with result from task 362 parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:59 proc_lsst.multi:146 [INFO] found job 17014809 in provider astro proc_lsst.multi INFO: found job 17014809 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/940665f7-f210-4da1-88f0-dc4db22722a9_isrForCrosstalkSources_899118_5.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/940665f7-f210-4da1-88f0-dc4db22722a9_isrForCrosstalkSources_899118_5.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry addcd5c9ebf99e28e8b7fc9317288dce with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/68594bb6-bfe0-4192-aeef-10f7bc796322_isrForCrosstalkSources_899117_21.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/68594bb6-bfe0-4192-aeef-10f7bc796322_isrForCrosstalkSources_899117_21.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 128986b998d254f861914491f8529118 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e5d716e-223b-4b57-b844-226daca72d37_isrForCrosstalkSources_899016_23.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/1e5d716e-223b-4b57-b844-226daca72d37_isrForCrosstalkSources_899016_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:59 proc_lsst.multi:146 [INFO] found job 17014810 in provider astro proc_lsst.multi INFO: found job 17014810 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76e37fa7898d55b91d6b0cb33f6ee2fb with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b9e1d3e6-cdd2-4a2a-9acd-a6deb5d88668_isrForCrosstalkSources_899016_25.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b9e1d3e6-cdd2-4a2a-9acd-a6deb5d88668_isrForCrosstalkSources_899016_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:59 proc_lsst.multi:146 [INFO] found job 17014812 in provider astro proc_lsst.multi INFO: found job 17014812 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:20:59 proc_lsst.multi:146 [INFO] found job 17014865 in provider astro proc_lsst.multi INFO: found job 17014865 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014866 in provider astro proc_lsst.multi INFO: found job 17014866 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 913d97f4abbf6f295f1b5465df91b6f1 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54a14e7a-3168-4390-beeb-0aeaea9bf360_isrForCrosstalkSources_899016_17.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54a14e7a-3168-4390-beeb-0aeaea9bf360_isrForCrosstalkSources_899016_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014867 in provider astro proc_lsst.multi INFO: found job 17014867 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014868 in provider astro proc_lsst.multi INFO: found job 17014868 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014869 in provider astro proc_lsst.multi INFO: found job 17014869 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014870 in provider astro proc_lsst.multi INFO: found job 17014870 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014871 in provider astro proc_lsst.multi INFO: found job 17014871 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014789,17014790,17014791,17014792,17014793,17014794,17014795,17014796,17014797,17014798,17014799,17014800,17014801,17014802,17014803,17014804,17014805,17014806,17014807,17014808,17014809,17014810,17014812,17014865,17014866,17014867,17014868,17014869,17014870,17014871' parsl.providers.slurm.slurm DEBUG: squeue returned 17014871 R 17014869 R 17014870 R 17014866 R 17014867 R 17014868 R 17014865 R 17014812 R 17014809 R 17014810 R 17014808 R 17014804 R 17014805 R 17014806 R 17014807 R 17014800 R 17014801 R 17014802 R 17014803 R 17014798 R 17014799 R 17014796 R 17014797 R 17014792 R 17014793 R 17014794 R 17014795 R 17014789 R 17014790 R 17014791 R parsl.providers.slurm.slurm DEBUG: Updating job 17014871 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014869 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014870 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014866 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014867 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014868 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014865 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014807 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014791 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:00 proc_lsst.multi:146 [INFO] found job 17014872 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014872 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014873 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014873 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014875 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014875 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014876 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014876 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014877 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014877 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014872,17014873,17014875,17014876,17014877' parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3ecd3cea071639eeb1c437429570558 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/45cce674-ce12-4fbc-80c3-c6118e5ee168_isrForCrosstalkSources_899117_20.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/45cce674-ce12-4fbc-80c3-c6118e5ee168_isrForCrosstalkSources_899117_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014877 R 17014875 R 17014876 R 17014872 R 17014873 R parsl.providers.slurm.slurm DEBUG: Updating job 17014877 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014875 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014876 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014872 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014873 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014878 in provider escience_a40 proc_lsst.multi INFO: found job 17014878 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014879 in provider escience_a40 proc_lsst.multi INFO: found job 17014879 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d057917a9f9b733a25e2ebf90136b604 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f7cc92f1-45e3-4ea5-b4e9-8d6cf44bb192_isrForCrosstalkSources_899118_40.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f7cc92f1-45e3-4ea5-b4e9-8d6cf44bb192_isrForCrosstalkSources_899118_40.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52edf6f83eeca816e13438bcf0418029 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2b12f40e-f93c-4ae7-8ced-759a46663fe0_isrForCrosstalkSources_899014_8.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/2b12f40e-f93c-4ae7-8ced-759a46663fe0_isrForCrosstalkSources_899014_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014880 in provider escience_a40 proc_lsst.multi INFO: found job 17014880 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b1eeb1730e18422345101497006238b with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b45cc4bd-602c-4070-927b-2376d2da1b5d_isrForCrosstalkSources_899016_20.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/b45cc4bd-602c-4070-927b-2376d2da1b5d_isrForCrosstalkSources_899016_20.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a08325180f29143f6676a9eee21b1585 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e3f7ece9-48c0-4f69-857b-8ec7a41c7bd0_isrForCrosstalkSources_899118_51.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/e3f7ece9-48c0-4f69-857b-8ec7a41c7bd0_isrForCrosstalkSources_899118_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:01 proc_lsst.multi:146 [INFO] found job 17014897 in provider escience_a40 proc_lsst.multi INFO: found job 17014897 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a90f91b59c7fa30e21bf2b1d1b6c7a3 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/eda664ab-92da-4125-b023-5838d3de5df1_isrForCrosstalkSources_899118_23.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/eda664ab-92da-4125-b023-5838d3de5df1_isrForCrosstalkSources_899118_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014898 in provider escience_a40 proc_lsst.multi INFO: found job 17014898 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6d6bff50500e0523b82dbf5250e2011 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/365d055d-3469-4cac-a582-ff0176ea01f0_isrForCrosstalkSources_899118_22.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/365d055d-3469-4cac-a582-ff0176ea01f0_isrForCrosstalkSources_899118_22.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 221e304d72ffeb3e55f8ac9bf9a46c44 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e673898e-77a1-4f1a-a1c4-a5117887e639_isrForCrosstalkSources_899117_5.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/e673898e-77a1-4f1a-a1c4-a5117887e639_isrForCrosstalkSources_899117_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014899 in provider escience_a40 proc_lsst.multi INFO: found job 17014899 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014900 in provider escience_a40 proc_lsst.multi INFO: found job 17014900 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014901 in provider escience_a40 proc_lsst.multi INFO: found job 17014901 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014902 in provider escience_a40 proc_lsst.multi INFO: found job 17014902 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014903 in provider escience_a40 proc_lsst.multi INFO: found job 17014903 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dff3200a45e4b937eae0260c3f1cf0b8 with result from task 343 parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ad38e85b-0a7a-4a8e-903f-13dc24aac2bc_isrForCrosstalkSources_899016_29.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/ad38e85b-0a7a-4a8e-903f-13dc24aac2bc_isrForCrosstalkSources_899016_29.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:02 proc_lsst.multi:146 [INFO] found job 17014904 in provider escience_a40 proc_lsst.multi INFO: found job 17014904 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014905 in provider escience_a40 proc_lsst.multi INFO: found job 17014905 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56016e23555bee6edb81046a8a98f637 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/45d46e0c-317f-4c35-bda5-58380cc0e63a_isrForCrosstalkSources_899118_50.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/45d46e0c-317f-4c35-bda5-58380cc0e63a_isrForCrosstalkSources_899118_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014906 in provider escience_a40 proc_lsst.multi INFO: found job 17014906 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a79aa101210756119aa931a01736b941 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1c7881c-96d3-43f9-a3c5-01515ed9af6e_isrForCrosstalkSources_899015_54.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899015/a1c7881c-96d3-43f9-a3c5-01515ed9af6e_isrForCrosstalkSources_899015_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014907 in provider escience_a40 proc_lsst.multi INFO: found job 17014907 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014908 in provider escience_a40 proc_lsst.multi INFO: found job 17014908 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014909 in provider escience_a40 proc_lsst.multi INFO: found job 17014909 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014910 in provider escience_a40 proc_lsst.multi INFO: found job 17014910 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:03 proc_lsst.multi:146 [INFO] found job 17014911 in provider escience_a40 proc_lsst.multi INFO: found job 17014911 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING 2024-03-13 13:21:04 proc_lsst.multi:146 [INFO] found job 17014912 in provider escience_a40 proc_lsst.multi INFO: found job 17014912 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17014878,17014879,17014880,17014897,17014898,17014899,17014900,17014901,17014902,17014903,17014904,17014905,17014906,17014907,17014908,17014909,17014910,17014911,17014912' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 022dff3bbb0fb6c357e4e526b60d04b4 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1565105a-8237-46a0-ab2f-8bc6c083c20c_isrForCrosstalkSources_899119_61.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/1565105a-8237-46a0-ab2f-8bc6c083c20c_isrForCrosstalkSources_899119_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17014911 R 17014912 R 17014909 R 17014910 R 17014908 R 17014907 R 17014904 R 17014905 R 17014906 R 17014902 R 17014903 R 17014899 R 17014900 R 17014901 R 17014897 R 17014898 R 17014880 R 17014878 R 17014879 R parsl.providers.slurm.slurm DEBUG: Updating job 17014911 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014912 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014909 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014910 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014908 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014907 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014904 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014905 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014906 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014902 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014903 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014899 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014900 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014901 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014897 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014898 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014880 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014878 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17014879 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 55, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 55/0 running/pending blocks, and 56 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23379652658256 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 55, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 55/0 running/pending blocks, and 56 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23379652658256 parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 237fdc6835c575808b1b1cdc55f9b8d3 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a561ccd-0c79-437c-b10f-5306243cc265_isrForCrosstalkSources_899119_46.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899119/4a561ccd-0c79-437c-b10f-5306243cc265_isrForCrosstalkSources_899119_46.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de9869604918946fb501de7c66875ba with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/986bd71f-1265-4e33-abb1-fcbe754694b2_isrForCrosstalkSources_899117_24.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/986bd71f-1265-4e33-abb1-fcbe754694b2_isrForCrosstalkSources_899117_24.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af273dcfde85c888019ce40ce60940ce with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/67b2e673-3c03-4861-ab79-c32894210b53_isrForCrosstalkSources_899014_15.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899014/67b2e673-3c03-4861-ab79-c32894210b53_isrForCrosstalkSources_899014_15.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49b644ff59c93b73b684b64e67a8d2e3 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f9868757-22c6-422b-9a54-26ccc42064e1_isrForCrosstalkSources_899118_9.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899118/f9868757-22c6-422b-9a54-26ccc42064e1_isrForCrosstalkSources_899118_9.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a974f515de6e42655120dca77a7c18e8 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77add4c4-904c-4782-a1ca-47a73640ba40_isrForCrosstalkSources_899117_2.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899117/77add4c4-904c-4782-a1ca-47a73640ba40_isrForCrosstalkSources_899117_2.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd3d31a28fe40f9af358a63b9d9a2bf9 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54df4f09-8aa7-467e-92e0-f961b71dda90_isrForCrosstalkSources_899016_21.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/54df4f09-8aa7-467e-92e0-f961b71dda90_isrForCrosstalkSources_899016_21.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91a27b89c1ebd54a52aab37564c99361 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/50db53df-fbbe-4572-94c4-4507257f12ee_isrForCrosstalkSources_899016_52.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/50db53df-fbbe-4572-94c4-4507257f12ee_isrForCrosstalkSources_899016_52.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d2345730bdbb1dbca588b3639b5c66f with result from task 313 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/52e005c1-eb0b-43f6-aa0c-fc5b7123b6bc_isrForCrosstalkSources_899016_47.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/submit/DEEP/20190928/B1a/science#step0/20240313T131829Z/logs/isrForCrosstalkSources/899016/52e005c1-eb0b-43f6-aa0c-fc5b7123b6bc_isrForCrosstalkSources_899016_47.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=55, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 55 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e2132e77016 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e2132e77016 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 802d94aff261 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 802d94aff261 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb8054c3dd5b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb8054c3dd5b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f78b11f7468 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f78b11f7468 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a1209344ecfb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a1209344ecfb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2677984d264 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2677984d264 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a69414966b55 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a69414966b55 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17fed16687be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17fed16687be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f4c481ea7bd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f4c481ea7bd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2af622048cf8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2af622048cf8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d21d32ffb17b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d21d32ffb17b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a952aca1c46b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a952aca1c46b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 69ae49a8a3c1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 69ae49a8a3c1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81e913088f7f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81e913088f7f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 83c332d67c71 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 83c332d67c71 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29fdf94e6b72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29fdf94e6b72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfa096f04ac8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfa096f04ac8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 82f958ad281f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 82f958ad281f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 27d804155d3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 27d804155d3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43f45c078bd6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43f45c078bd6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 92c1aee6a305 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 92c1aee6a305 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4cb9de863cd9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4cb9de863cd9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f515bcf86a5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f515bcf86a5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 363c7069fc1b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 363c7069fc1b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2fa82149a4d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2fa82149a4d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0a375700933c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0a375700933c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ae431e55ad1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ae431e55ad1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48e1b5a9f7cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48e1b5a9f7cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33f896811099 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33f896811099 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d81d043c2472 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d81d043c2472 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d6e504206b18 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d6e504206b18 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5691f0d776a3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5691f0d776a3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6e8ddf634ee8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6e8ddf634ee8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 227f71e59597 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 227f71e59597 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c148523ed8e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c148523ed8e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9da84950f0b2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9da84950f0b2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f9fddc8103e4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f9fddc8103e4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd72e37dde4f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd72e37dde4f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a64ea9cb8a51 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a64ea9cb8a51 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfce9c33cf57 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfce9c33cf57 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 22c17bb8dc5f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 22c17bb8dc5f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e6410dd1bb8b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e6410dd1bb8b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8ff11b3a7fc2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8ff11b3a7fc2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cd3909f6e6ec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cd3909f6e6ec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 975ccef6dd58 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 975ccef6dd58 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c25a4401285e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c25a4401285e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: adfb7bc19109 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: adfb7bc19109 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df68c5cb3d5b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df68c5cb3d5b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 089f623a7ee3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 089f623a7ee3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6f45b3480ddb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6f45b3480ddb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fa03323874f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fa03323874f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc76a6a623ae parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc76a6a623ae parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9d9c895d3b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9d9c895d3b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 817d906f9dad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 817d906f9dad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81a5821c5592 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81a5821c5592 2024-03-13 13:21:04 proc_lsst.multi:146 [INFO] found job 17014908 in provider escience_a40 proc_lsst.multi INFO: found job 17014908 in provider escience_a40 2024-03-13 13:21:04 proc_lsst.multi:201 [INFO] cancelling 17014908 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014908 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014912 in provider escience_a40 proc_lsst.multi INFO: found job 17014912 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014912 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014912 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014911 in provider escience_a40 proc_lsst.multi INFO: found job 17014911 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014911 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014911 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014907 in provider escience_a40 proc_lsst.multi INFO: found job 17014907 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014907 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014907 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014905 in provider escience_a40 proc_lsst.multi INFO: found job 17014905 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014905 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014905 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014906 in provider escience_a40 proc_lsst.multi INFO: found job 17014906 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014906 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014906 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014909 in provider escience_a40 proc_lsst.multi INFO: found job 17014909 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014909 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014909 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014910 in provider escience_a40 proc_lsst.multi INFO: found job 17014910 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014910 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014910 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014878 in provider escience_a40 proc_lsst.multi INFO: found job 17014878 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014878 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014878 on provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:146 [INFO] found job 17014904 in provider escience_a40 proc_lsst.multi INFO: found job 17014904 in provider escience_a40 2024-03-13 13:21:05 proc_lsst.multi:201 [INFO] cancelling 17014904 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014904 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014867 in provider astro proc_lsst.multi INFO: found job 17014867 in provider astro 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014867 on provider astro proc_lsst.multi INFO: cancelling 17014867 on provider astro 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014880 in provider escience_a40 proc_lsst.multi INFO: found job 17014880 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014880 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014880 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014899 in provider escience_a40 proc_lsst.multi INFO: found job 17014899 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014899 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014899 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014898 in provider escience_a40 proc_lsst.multi INFO: found job 17014898 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014898 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014898 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014900 in provider escience_a40 proc_lsst.multi INFO: found job 17014900 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014900 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014900 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014902 in provider escience_a40 proc_lsst.multi INFO: found job 17014902 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014902 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014902 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014897 in provider escience_a40 proc_lsst.multi INFO: found job 17014897 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014897 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014897 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014901 in provider escience_a40 proc_lsst.multi INFO: found job 17014901 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014901 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014901 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014879 in provider escience_a40 proc_lsst.multi INFO: found job 17014879 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014879 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014879 on provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:146 [INFO] found job 17014903 in provider escience_a40 proc_lsst.multi INFO: found job 17014903 in provider escience_a40 2024-03-13 13:21:06 proc_lsst.multi:201 [INFO] cancelling 17014903 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014903 on provider escience_a40 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014875 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014875 in provider escience_rtx6k 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014875 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014875 on provider escience_rtx6k 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014796 in provider astro proc_lsst.multi INFO: found job 17014796 in provider astro 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014796 on provider astro proc_lsst.multi INFO: cancelling 17014796 on provider astro 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014877 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014877 in provider escience_rtx6k 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014877 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014877 on provider escience_rtx6k 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 86094 in provider local proc_lsst.multi INFO: found job 86094 in provider local 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 86094 on provider local proc_lsst.multi INFO: cancelling 86094 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 86094 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014790 in provider astro proc_lsst.multi INFO: found job 17014790 in provider astro 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014790 on provider astro proc_lsst.multi INFO: cancelling 17014790 on provider astro 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014797 in provider astro proc_lsst.multi INFO: found job 17014797 in provider astro 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014797 on provider astro proc_lsst.multi INFO: cancelling 17014797 on provider astro 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014792 in provider astro proc_lsst.multi INFO: found job 17014792 in provider astro 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014792 on provider astro proc_lsst.multi INFO: cancelling 17014792 on provider astro 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014808 in provider astro proc_lsst.multi INFO: found job 17014808 in provider astro 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014808 on provider astro proc_lsst.multi INFO: cancelling 17014808 on provider astro 2024-03-13 13:21:07 proc_lsst.multi:146 [INFO] found job 17014873 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014873 in provider escience_rtx6k 2024-03-13 13:21:07 proc_lsst.multi:201 [INFO] cancelling 17014873 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014873 on provider escience_rtx6k 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014802 in provider astro proc_lsst.multi INFO: found job 17014802 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014802 on provider astro proc_lsst.multi INFO: cancelling 17014802 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014795 in provider astro proc_lsst.multi INFO: found job 17014795 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014795 on provider astro proc_lsst.multi INFO: cancelling 17014795 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014803 in provider astro proc_lsst.multi INFO: found job 17014803 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014803 on provider astro proc_lsst.multi INFO: cancelling 17014803 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014794 in provider astro proc_lsst.multi INFO: found job 17014794 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014794 on provider astro proc_lsst.multi INFO: cancelling 17014794 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014793 in provider astro proc_lsst.multi INFO: found job 17014793 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014793 on provider astro proc_lsst.multi INFO: cancelling 17014793 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014806 in provider astro proc_lsst.multi INFO: found job 17014806 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014806 on provider astro proc_lsst.multi INFO: cancelling 17014806 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014812 in provider astro proc_lsst.multi INFO: found job 17014812 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014812 on provider astro proc_lsst.multi INFO: cancelling 17014812 on provider astro 2024-03-13 13:21:08 proc_lsst.multi:146 [INFO] found job 17014801 in provider astro proc_lsst.multi INFO: found job 17014801 in provider astro 2024-03-13 13:21:08 proc_lsst.multi:201 [INFO] cancelling 17014801 on provider astro proc_lsst.multi INFO: cancelling 17014801 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014809 in provider astro proc_lsst.multi INFO: found job 17014809 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014809 on provider astro proc_lsst.multi INFO: cancelling 17014809 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014791 in provider astro proc_lsst.multi INFO: found job 17014791 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014791 on provider astro proc_lsst.multi INFO: cancelling 17014791 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014871 in provider astro proc_lsst.multi INFO: found job 17014871 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014871 on provider astro proc_lsst.multi INFO: cancelling 17014871 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014800 in provider astro proc_lsst.multi INFO: found job 17014800 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014800 on provider astro proc_lsst.multi INFO: cancelling 17014800 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014870 in provider astro proc_lsst.multi INFO: found job 17014870 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014870 on provider astro proc_lsst.multi INFO: cancelling 17014870 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014869 in provider astro proc_lsst.multi INFO: found job 17014869 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014869 on provider astro proc_lsst.multi INFO: cancelling 17014869 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014799 in provider astro proc_lsst.multi INFO: found job 17014799 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014799 on provider astro proc_lsst.multi INFO: cancelling 17014799 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014805 in provider astro proc_lsst.multi INFO: found job 17014805 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014805 on provider astro proc_lsst.multi INFO: cancelling 17014805 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014804 in provider astro proc_lsst.multi INFO: found job 17014804 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014804 on provider astro proc_lsst.multi INFO: cancelling 17014804 on provider astro 2024-03-13 13:21:09 proc_lsst.multi:146 [INFO] found job 17014798 in provider astro proc_lsst.multi INFO: found job 17014798 in provider astro 2024-03-13 13:21:09 proc_lsst.multi:201 [INFO] cancelling 17014798 on provider astro proc_lsst.multi INFO: cancelling 17014798 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014872 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014872 in provider escience_rtx6k 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014872 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014872 on provider escience_rtx6k 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014866 in provider astro proc_lsst.multi INFO: found job 17014866 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014866 on provider astro proc_lsst.multi INFO: cancelling 17014866 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014865 in provider astro proc_lsst.multi INFO: found job 17014865 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014865 on provider astro proc_lsst.multi INFO: cancelling 17014865 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014868 in provider astro proc_lsst.multi INFO: found job 17014868 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014868 on provider astro proc_lsst.multi INFO: cancelling 17014868 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014789 in provider astro proc_lsst.multi INFO: found job 17014789 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014789 on provider astro proc_lsst.multi INFO: cancelling 17014789 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014810 in provider astro proc_lsst.multi INFO: found job 17014810 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014810 on provider astro proc_lsst.multi INFO: cancelling 17014810 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014807 in provider astro proc_lsst.multi INFO: found job 17014807 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014807 on provider astro proc_lsst.multi INFO: cancelling 17014807 on provider astro 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014876 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014876 in provider escience_rtx6k 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014876 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014876 on provider escience_rtx6k parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-13 13:21:10 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-13 13:21:10 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17014789', '17014790', '17014791', '17014792', '17014793', '17014794', '17014795', '17014796', '17014797', '17014798', '17014799', '17014800', '17014801', '17014802', '17014803', '17014804', '17014805', '17014806', '17014807', '17014808', '17014809', '17014810', '17014812', '17014865', '17014866', '17014867', '17014868', '17014869', '17014870', '17014871', '17014878', '17014879', '17014880', '17014897', '17014898', '17014899', '17014900', '17014901', '17014902', '17014903', '17014904', '17014905', '17014906', '17014907', '17014908', '17014909', '17014910', '17014911', '17014912', '17014872', '17014873', '17014875', '17014876', '17014877', '86094'] proc_lsst.multi INFO: new jobs since last cancel ['17014789', '17014790', '17014791', '17014792', '17014793', '17014794', '17014795', '17014796', '17014797', '17014798', '17014799', '17014800', '17014801', '17014802', '17014803', '17014804', '17014805', '17014806', '17014807', '17014808', '17014809', '17014810', '17014812', '17014865', '17014866', '17014867', '17014868', '17014869', '17014870', '17014871', '17014878', '17014879', '17014880', '17014897', '17014898', '17014899', '17014900', '17014901', '17014902', '17014903', '17014904', '17014905', '17014906', '17014907', '17014908', '17014909', '17014910', '17014911', '17014912', '17014872', '17014873', '17014875', '17014876', '17014877', '86094'] 2024-03-13 13:21:10 proc_lsst.multi:146 [INFO] found job 17014789 in provider astro proc_lsst.multi INFO: found job 17014789 in provider astro 2024-03-13 13:21:10 proc_lsst.multi:201 [INFO] cancelling 17014789 on provider astro proc_lsst.multi INFO: cancelling 17014789 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014790 in provider astro proc_lsst.multi INFO: found job 17014790 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014790 on provider astro proc_lsst.multi INFO: cancelling 17014790 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014791 in provider astro proc_lsst.multi INFO: found job 17014791 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014791 on provider astro proc_lsst.multi INFO: cancelling 17014791 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014792 in provider astro proc_lsst.multi INFO: found job 17014792 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014792 on provider astro proc_lsst.multi INFO: cancelling 17014792 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014793 in provider astro proc_lsst.multi INFO: found job 17014793 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014793 on provider astro proc_lsst.multi INFO: cancelling 17014793 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014794 in provider astro proc_lsst.multi INFO: found job 17014794 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014794 on provider astro proc_lsst.multi INFO: cancelling 17014794 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014795 in provider astro proc_lsst.multi INFO: found job 17014795 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014795 on provider astro proc_lsst.multi INFO: cancelling 17014795 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014796 in provider astro proc_lsst.multi INFO: found job 17014796 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014796 on provider astro proc_lsst.multi INFO: cancelling 17014796 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014797 in provider astro proc_lsst.multi INFO: found job 17014797 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014797 on provider astro proc_lsst.multi INFO: cancelling 17014797 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014798 in provider astro proc_lsst.multi INFO: found job 17014798 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014798 on provider astro proc_lsst.multi INFO: cancelling 17014798 on provider astro 2024-03-13 13:21:11 proc_lsst.multi:146 [INFO] found job 17014799 in provider astro proc_lsst.multi INFO: found job 17014799 in provider astro 2024-03-13 13:21:11 proc_lsst.multi:201 [INFO] cancelling 17014799 on provider astro proc_lsst.multi INFO: cancelling 17014799 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014800 in provider astro proc_lsst.multi INFO: found job 17014800 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014800 on provider astro proc_lsst.multi INFO: cancelling 17014800 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014801 in provider astro proc_lsst.multi INFO: found job 17014801 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014801 on provider astro proc_lsst.multi INFO: cancelling 17014801 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014802 in provider astro proc_lsst.multi INFO: found job 17014802 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014802 on provider astro proc_lsst.multi INFO: cancelling 17014802 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014803 in provider astro proc_lsst.multi INFO: found job 17014803 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014803 on provider astro proc_lsst.multi INFO: cancelling 17014803 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014804 in provider astro proc_lsst.multi INFO: found job 17014804 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014804 on provider astro proc_lsst.multi INFO: cancelling 17014804 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014805 in provider astro proc_lsst.multi INFO: found job 17014805 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014805 on provider astro proc_lsst.multi INFO: cancelling 17014805 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014806 in provider astro proc_lsst.multi INFO: found job 17014806 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014806 on provider astro proc_lsst.multi INFO: cancelling 17014806 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014807 in provider astro proc_lsst.multi INFO: found job 17014807 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014807 on provider astro proc_lsst.multi INFO: cancelling 17014807 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014808 in provider astro proc_lsst.multi INFO: found job 17014808 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014808 on provider astro proc_lsst.multi INFO: cancelling 17014808 on provider astro 2024-03-13 13:21:12 proc_lsst.multi:146 [INFO] found job 17014809 in provider astro proc_lsst.multi INFO: found job 17014809 in provider astro 2024-03-13 13:21:12 proc_lsst.multi:201 [INFO] cancelling 17014809 on provider astro proc_lsst.multi INFO: cancelling 17014809 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014810 in provider astro proc_lsst.multi INFO: found job 17014810 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014810 on provider astro proc_lsst.multi INFO: cancelling 17014810 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014812 in provider astro proc_lsst.multi INFO: found job 17014812 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014812 on provider astro proc_lsst.multi INFO: cancelling 17014812 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014865 in provider astro proc_lsst.multi INFO: found job 17014865 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014865 on provider astro proc_lsst.multi INFO: cancelling 17014865 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014866 in provider astro proc_lsst.multi INFO: found job 17014866 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014866 on provider astro proc_lsst.multi INFO: cancelling 17014866 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014867 in provider astro proc_lsst.multi INFO: found job 17014867 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014867 on provider astro proc_lsst.multi INFO: cancelling 17014867 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014868 in provider astro proc_lsst.multi INFO: found job 17014868 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014868 on provider astro proc_lsst.multi INFO: cancelling 17014868 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014869 in provider astro proc_lsst.multi INFO: found job 17014869 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014869 on provider astro proc_lsst.multi INFO: cancelling 17014869 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014870 in provider astro proc_lsst.multi INFO: found job 17014870 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014870 on provider astro proc_lsst.multi INFO: cancelling 17014870 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014871 in provider astro proc_lsst.multi INFO: found job 17014871 in provider astro 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014871 on provider astro proc_lsst.multi INFO: cancelling 17014871 on provider astro 2024-03-13 13:21:13 proc_lsst.multi:146 [INFO] found job 17014878 in provider escience_a40 proc_lsst.multi INFO: found job 17014878 in provider escience_a40 2024-03-13 13:21:13 proc_lsst.multi:201 [INFO] cancelling 17014878 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014878 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014879 in provider escience_a40 proc_lsst.multi INFO: found job 17014879 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014879 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014879 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014880 in provider escience_a40 proc_lsst.multi INFO: found job 17014880 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014880 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014880 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014897 in provider escience_a40 proc_lsst.multi INFO: found job 17014897 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014897 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014897 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014898 in provider escience_a40 proc_lsst.multi INFO: found job 17014898 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014898 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014898 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014899 in provider escience_a40 proc_lsst.multi INFO: found job 17014899 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014899 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014899 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014900 in provider escience_a40 proc_lsst.multi INFO: found job 17014900 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014900 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014900 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014901 in provider escience_a40 proc_lsst.multi INFO: found job 17014901 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014901 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014901 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014902 in provider escience_a40 proc_lsst.multi INFO: found job 17014902 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014902 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014902 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014903 in provider escience_a40 proc_lsst.multi INFO: found job 17014903 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014903 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014903 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014904 in provider escience_a40 proc_lsst.multi INFO: found job 17014904 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014904 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014904 on provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:146 [INFO] found job 17014905 in provider escience_a40 proc_lsst.multi INFO: found job 17014905 in provider escience_a40 2024-03-13 13:21:14 proc_lsst.multi:201 [INFO] cancelling 17014905 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014905 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014906 in provider escience_a40 proc_lsst.multi INFO: found job 17014906 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014906 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014906 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014907 in provider escience_a40 proc_lsst.multi INFO: found job 17014907 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014907 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014907 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014908 in provider escience_a40 proc_lsst.multi INFO: found job 17014908 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014908 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014908 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014909 in provider escience_a40 proc_lsst.multi INFO: found job 17014909 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014909 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014909 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014910 in provider escience_a40 proc_lsst.multi INFO: found job 17014910 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014910 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014910 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014911 in provider escience_a40 proc_lsst.multi INFO: found job 17014911 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014911 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014911 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014912 in provider escience_a40 proc_lsst.multi INFO: found job 17014912 in provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014912 on provider escience_a40 proc_lsst.multi INFO: cancelling 17014912 on provider escience_a40 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014872 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014872 in provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014872 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014872 on provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014873 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014873 in provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014873 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014873 on provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014875 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014875 in provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014875 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014875 on provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014876 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014876 in provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014876 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014876 on provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:146 [INFO] found job 17014877 in provider escience_rtx6k proc_lsst.multi INFO: found job 17014877 in provider escience_rtx6k 2024-03-13 13:21:15 proc_lsst.multi:201 [INFO] cancelling 17014877 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17014877 on provider escience_rtx6k 2024-03-13 13:21:16 proc_lsst.multi:146 [INFO] found job 86094 in provider local proc_lsst.multi INFO: found job 86094 in provider local 2024-03-13 13:21:16 proc_lsst.multi:201 [INFO] cancelling 86094 on provider local proc_lsst.multi INFO: cancelling 86094 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 86094 parsl.providers.local.local WARNING: Failed to kill PID: 86094 and child processes on local 2024-03-13 13:21:17 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 153.5811 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20190928_B1a_science#step0_20240313T131829Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 153.5873 seconds; current memory usage: 0.275 Gibyte, delta: 0.002 Gibyte, peak delta: 0.004 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 165.6680 seconds; current memory usage: 0.275 Gibyte, delta: 0.138 Gibyte, peak delta: 0.140 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.277 Gibyte (main), 0.277 Gibyte (largest child process) Run Id: None Run Name: DEEP_20190928_B1a_science#step0_20240313T131829Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m47.568s user 0m35.784s sys 0m16.320s